Django: it is sometimes faster to use built in python function for model filtering

In a heavy loop, you might want to change django model filtering to built in python filter function.

Consider this

from myStuff.models import MyModel
myModelItems = MyModel.objects.filter(pk__gt = 15)

for x in range(1, 500):
   myModelItems.filter(pk__gte = random.randint(15, 35))[0].pk

now this is of course bogus code, but it shows the point:

• we have a loop that will execute code many times
• we have same model items selected many times

In this case it is better to use built in python function:

from myStuff.models import MyModel
myModelItems = MyModel.objects.filter(pk__gt = 15)

for x in range(1, 500):
   filter(lambda x : x. pk >= random.randint(15, 35), myModelItems)[0].pk

Because django-filter will return new objects and it will have to hit the database every time, while built in function will return the same object and if it was already used, it will not have to reconsult the database.

Definning first segment as a and second as b gives these results:

timeit(a, 5)
#1.0679619312286377
timeit(b, 5)
#0.19536900520324707

and it gets faster and faster if range is increased.

5 Vegan Super Foods You Will Find in Any Supermarket

Here is the post about vegan super foods you should eat everyday. The best part is — they are really cheap and you will have no trouble finding them in your closest supermarket!

B12 is not the only vitamin vegans will not get naturally with their food

Recently I’ve published an article about vitamins called B12 is not the only vitamin vegans will not get naturally with their food, it is about vitamin D, where to get it and why we need it.

Vegan Diet for Vegan Athlete

For the past few months I’ve been working on my project for vegans called Vegan Muscles – Vegan Diet for Vegan Athlete.

In the future Vegan Muscles will create personalized vegan diet based on activity level and goals. It will calculate daily intake for carbs, proteins and fats as well as every amino acid, mineral and vitamin and suggest weekly recipes.

It is still in quite early development mode so it is still manual, but more features are coming really soon! As for now you can use Vegan Muscles to see if you are not missing any vital nutrients, or read vegan blog full of tips for vegan nutrition. It is quite young blog too, but articles have been accepted by vegan community quite well.

Go Vegan – Stay Active!

Hackers & Painters review

Hackers and Painters : Big Ideas from the Computer Age is a collection of essays written by Paul Graham. I think you can find them all on his personal web page here:

http://www.paulgraham.com/

If you are not planning on buying this book, I at least highly recommend to read the essays online. They are awesome, but lets get ourselves there. After all, I’m writing a review.

Paul Graham

First lets talk about Paul Graham himself. Not in a sense of his bio, you can find it on his website. But about him as a person. Of course, he might be completely different, but this is how I see him through the book and online essays.

He is a hacker, LISP evangelist, OOP critic and in fact a critic in general. That is the most important. He is a critic and skeptic of all the things.

He is also very opinionated and together with his skepticism, it makes this book really great. I could easily say that he has a strong opinion about any topic. The book is about computers in general, but Paul will take a chance to state his opinion not only on that. One example could be his view on social issues. We can all remember how jobless young people flooded the streets of NYC in 2012 and later all over the world (with the 99% bullshit). What they were protesting against is what Paul Graham states as a sign of healthy society. And he is completely right.

One of his chapters is called “What You Can’t Say” and throughout the book he always says what is the opposite of what the majority would believe. Sometimes I would agree with him and read with a satisfied smile on my face seeing how he crushes those that “know wrong”. Sometimes I would disagree and would read with other kind of pleasure thinking of arguments I could use against him.

The Book

The book is written in a very great style. It is almost as having a real conversation with author, not just reading. It got me smiling, laughing and grinding my teeth. But most importantly it got me thinking numerous times. I’ve read it mostly on the bus and at least half of the time I was not actually reading. I was looking through the window playing with ideas and memories that were generated by the words written in the pages. If this is not a sign of awesome book, I don’t know what is.

Who should read it? Well, hackers of course. Even if essays are quite old (the book was published in 2004 and essays written even a longer time ago), the great ideas are time-proof and the book is full of them. It will inspire, motivate and overall make you a better programmer.

Anyone else? Probably those that have to deal with hackers on the daily basis. You know: wives, girlfriends, husbands, boyfriends. This book reveals how the brains of a hacker work. Not all chapters of course (programming languages are probably not very interesting to those who don’t code), but some of them are highly recommended. So if you have a hard time understanding your life partner who happens to be a hacker, give it a go. I’d recommend reading first 7 chapters.

But wait, you might say, there is another kind of people that deal with hackers on the daily basis. Technical people that don’t know anything about technology. You know, most of the project managers and software analysts. To be fair, even most of programmers. Those that should be good at understanding how things work down there but really don’t. Should they read it? Without a doubt. If they would listen to the words in this book, real hackers will start to hate them less, projects will fail less and in result we all would deliver a better software.

Generating Tiles With QGIS

Finally I’ve committed to start a project for generating Tiles with Qgis, if you’d like to fallow, here it is on GitHub:

https://github.com/mykk/quantumTiles

It only has a very hardcoded file I used to generate files outside Qgis, but eventually I hope it’ll turn to a useful Qgis plugin.

Learn New Language Every Year

Why not INTERCAL? Ever heard of Mortgage Code? It is writing code so bizarre and complex that only you can (hardly) understand what’s going on, thus your employer will be too feared to lay you off, keeping you safe while you’re on that loan.

Well INTERCAL takes it to the whole new level. Here is a fine paper on this programming language:

http://www.catb.org/~esr/intercal/paper.html

And here is a story behind it:

Alexander Garrett wrote a paper on INTERCAL for his Spring 1997 Programming Languages: Theory and Design class. Alexander writes:

The obvious choice was INTERCAL (I’m still quite surprised that I’m the only one who picked it — most people did Java??). Anyway, it was not favourably received…when [the professor] handed it back, he said, “Ah. I see you’re someone with a sense of humour. Unfortunately for you, I’m not.”

taken from here http://www.catb.org/~esr/intercal/

Have fun supporting your new investment!

Asserts are Awesome (and Exceptions certaintly don’t replace them)

Assert vs Exception

That actually is a quite common rant and it is usually won by exception. Even Jon Skeet thinks that assert could be replaced entirely by exception. Many people will read his opinion and just fallow it and that is unfortunate. I wish everyone would use assertions more. As I see them, they have completely different purpose from exceptions.

No Room for Assert?

First lets think of why people think assert can be replaced by exception. At first glance I need an assert to check if program is not in unexpected state.

For example, we expect that some argument will not be nullptr:

void myFunction(ImportantType* importantVariable)
{
   assert(importantVariable != nullptr);
   ...
}

or some function returns not nullptr

void myFunction()
{
   ...
   ImportantType* importantVariable = someImportantFunction();
   assert(importantVariable != nullptr);
   ...
}

or our result is valid

ImportantType* myFunction()
{
   ...
  assert(resultIsCorrect(someResult));
  return someResult;
}

But isn’t that risky? Using or returning value that we know is not correct? It might very well screw up other calculations and that will lead to many more problems. The best we can do here is stop execution before something worse happens and go to safety where we know what to do. This sounds like a job for exception. Isn’t it better to just throw rather than try to work with something we were not expecting?

After all doesn’t assertion mean “this shouldn’t have happened”? Seems like there is no room for assert…

Not a Chance! Asserts are Awesome!

OK, so where is assert better than exception? Well there are cases when I could not live without assert and exceptions just wouldn’t do the trick. And that is in private methods.

Lets take the simplest example possible. In public method if I can’t find a file, I should throw, but that shouldn’t always be the case with private methods, for example in some cases I could assume that file is empty. Where can I do it? Well, if everywhere where I use this private method, I wrap it in try catch, just to work like nothing has been done, I can very well do the same in the private method. Without exception swallowing information that something indeed went wrong.

Suppose you have some archives you want to load to your application. Some lines of it might be corrupt or even entire archive. So we need to take care of this, we don’t want to let some unpredictable data to enter our application. But we also don’t want application to crash just because it didn’t get some semi-important data. (Lets say we’re loading road network, if some of roads are corrupt it is better just to skip them, it’s not worth killing entire application or data load process, neither it is logical to add them if we know they will contain corrupt data) So, the code could look like this:

for (std::vector<std::string>::const_iterator it = files.begin(); it != files.end(); ++it)
{
   try
   {
      DataFile dataFile = OpenFile(*it);
      for (dataRow jt = dataFile.dataRowsBegin(); jt != dataFile.dataRowsEnd(); ++jt)
      {
         try
         {
            addData(*jt);
         }
         catch(Exception ex)
         {
            //something is wrong with data line... log something...
         }
      }
   }
   catch(Exception ex)
   {
      //something is wrong with the file... log something or do something else
   }
}

Well that sure looks ugly… We can move inner loop to another function, but just having try catch blocks inside a loop doesn’t look so well.

Another problem with it is that we might never notice that something is wrong in the first place unless we read logs and I don’t know anyone that looks at them unless something wrong happened. So it looks like we finally have somewhere to use assert — we can place it inside catch blocks, this way user wont be interrupted but on debug mode we will see that something did not go as expected right away!

But why do we need exceptions in the first place? We can simply assert inside local functions and declare a contract, that for corrupt file, it will log and then return empty dataFile and for corrupt line it will log and  just return.

Bottom Line

Don”t overlook assertions. They can help you spot bugs in your software much easier than exceptions. For me answer whether use exception or assert looks like this:

• In public methods exception wins. But I still place them to check arguments just that developer using my class (most likely it’s me) would be informed that he is doing something he is not supposed to.
• In private methods assert wins. Not all the time of course, but there is nothing I hate more than swallowed exceptions. All they do is hide bugs, while assert pops them out directly to your face.

Cartography as an Art

Just a reblog this time.

http://www.theatlanticcities.com/arts-and-lifestyle/2012/08/mapmaker-artist-or-programmer/3132/

Solo Developer: pros and cons

I am leaving my current position and decided to see what I’ve accomplished and learned for past two years. For all that time I’ve been a solo developer (well to be honest, I had an intern to help me out, but for quite short time anyway). Like everything in life, being a lonesome coder in this ruthless software world had both — pros and cons.

I think this was great and vital experience in my career. It allowed me to accomplish and learn a lot as I was responsible for absolutely everything. I wouldn’t change it for anything. Here is why:

• There is no one to turn to when you are alone. Where does that leave you? You have to find solutions completely on your own. That goes from choosing best algorithm, container, tools and languages to something like  finding out what was with that DLL problem on client machine. Thus being solo developer will, without any doubt, make you superb at problem solving.
• If you are willing to learn to be a better coder, you will. Use Stackoverflow, programmers.stackexchange, blogs and just try to use best solutions you find. Of course, no one is stopping you to do so with the team, but being alone you are your own boss and you’re totally free to experiment. I had no mentor, so what else could I do than to be my own mentor? It forced me to read books and that, combined with numerous resources on internet, is by far the best mentor you could ever have. Have an idea? Just browse for it!
• When working with someone else, your responsibilities are shared. You could be responsible for routing algorithms, while others work with map engine, database, etc. I had to know absolutely everything and without taking too much. Reading code becomes as easy as reading a book.
• I have learned what I needed with GIS from ground up. That might not sound very challenging, but before, I didn’t even had an slightest idea about any of that stuff. Hell, I didn’t even know it was called GIS! If I had been working with someone more experienced in this field, no doubt I wouldn’t have learned so much.
• I have written first unit tests, re-coded vital parts of code to drop copy-pasted code using design patterns, started using RAII, significantly increased performance for routing algorithms and included boost libraries. It’s not like I wouldn’t be able to do all that when I’m not alone, but again, I was my own boss and that gave me a lot of freedom to see what works and what doesn’t.

So what are the cons of being a solo developer? Well, firstly I think it depends on your views and experience you already have. If I wouldn’t have worked with great developers in the past to teach me and most importantly inspire me, I would have failed miserably.

But here is a single most important drawback: motivation — you have no one to share ideas. When you are stuck with some problem you only have monologue in your head instead of sharing it with others. I mean you can share it with others in the company, but will they be that interested in your whining? They have their own problems. At first it is nothing… But working alone for a year or longer can make you really miserable.

But non the less, as I’ve said, it was a really great experience and I don’t think I would have gained it if I wasn’t working alone. And now it is time to join a team and see what I have really learned during those years!