+972 52-548-6969
The value of parallelism
I said that I used the little script that I posted about for short tasks.
The reason for that is that it will run the files serially. This isn't very good if you want to get the maximum from your server. I actually wrote a multi threaded sql running that connects to a database and runs multiply queries in parallel. I'm using that for long queries (where long is thousands of files and the whole process takes hours).
Running it in parallel (and from a different computer than the one the server is running on) improved the performance by a factor of three (think about going from days to hours). It's so useful that I started to bulid an infra-structure around it.
What is basically does is to scan a directory (recursively) and put all the SQL files it finds in a queue, they are then run by a ThreadPool thread. I arrange it so that it must complete running all the files in a directory before it can move to the next one, so I can now arrange my scripts in directories and they will execute in the order of the directories names.
It's what I call a Me-Ware, it requires the task manager to close if you need to shut it down before it is done, the connection strings are hard coded, etc. Nevertheless, I find it incredibly useful.
In fact, I built the tool while waiting for the @#$% scripts to finish running :-)
Comments
Comment preview