Note To Self

replogo

ReportLab is full of different objects that you can place anywhere around the screen.  Following our previous code, we are making a list of elements that we want to draw onto a document.  ReportLab will handle all of the page breaks and lining things up but we can manipulate that a bit to help ourselves get a really clean looking PDF.  You can review the last two guides in the series by clicking these links, ReportLab and Django – Part 1 – The Set Up and a Basic Example, ReportLab and Django – Part 2 – Headers and Footers with Page Numbers.

The two most common elements to get your document to look nice are Paragraphs and Tables.  Paragraphs are, just as they sound, a paragraph of text that can contain some formatting to make it look nice.  It can even include images.  You can set a ParagraphStyle for the entire paragraph or put XML tags inside the Paragraph to get various styles throughout one object.

Paragraphs

A Paragraph is an element that spans the entire width of the area that it is given.  It holds text or images and will wrap when it reaches the end of the drawable area.  You can set a style for a Paragraph by creating a ParagraphStyle and applying it to your Paragraph.  ReportLab comes with a number of built in styles but I found that I preferred creating my own styles to give me more control.

In the example you can see that we first register some fonts.  Fonts are a very important feature of ReportLab because it allows you to use any TrueType Fonts you can find.  This is important because if you are going to be using any foreign language characters you’ll need a font that supports those characters.  If you don’t ReportLab will render them as black squares.

We then got the large amount of style sheets already created for us by ReportLab and then we add to it.  You can see that we add a new ParagraphStyle called ‘RightAlign’ that will use the Arial font and right align any text inside of it.  Simple enough.  To use it we do just like we did in the example in Part 2 but instead of using a prebuilt stylesheet like Heading1 we just call our own RightAlign.  Below is the example with the changed line highlighted.

There are a lot you can do within a Paragraph as well using inline tags.  If you wanted to make changes to a certain section of text and wanted to change it to be a different color, font, or size you can by using the <font> tags.  Adding an image is equally as simple by adding the <img> tag and specifying a file location.  Below is an example of a Paragraph with the same style we created above but with altered font and image added inline.

This creates a right aligned paragraph that has a bold red My followed by a normal Arial User Names and then a image of a smiley face with a height of 5.

Tables

Tables are the biggest help ever for getting things all lined up.  A Table is exactly like every other instance of a table you know.  It has rows and columns and you can specify their size and what is inside of them.  Tables by default will grow to contain whatever object is inside of them.  You can explicitly set the widths and heights of the columns and rows but your text/images can still overflow onto the outside of them.  Getting the widths and heights right is key.  Let’s make a basic table with some user data and set some table styles to make it look great.

This will create a table with all of our users on it, their usernames, and when they last logged in.  The table will be lined, that’s what the INNERGRID and BOX style does.  You can see that the table will span the entire page and that each column will take up a third of the document’s width.  You can customize these for varying column widths.  There are a lot of various customization options for tables.  Read them on the ReportLab docs.

I hope this was helpful.  I spent a lot of time figuring these things out as the documentation and examples are sparse.  If there are any problems you have that you want to know about let me know in the comments below.

Git-Logo-1788C

We recently moved our git repositories over to Amazon Web Services.  We ran into one issue with it which was that now our git requests needed to have a key pair attached.  The Internet was not very kind on explaining how to do this very well so I’m documenting it here.

The key pair you’ll want to use is the one generated for you by AWS.  It is the same one you use when you ssh into your AWS EC2 instances.  Take this key pair file, which I will call keypair.pem from now on, and we’ll need to move it and configure our git to use it.

  1. Copy the keypair.pem into your .ssh folder.
  2. Create/open your .ssh config file.
  3. In your config file
  4.  To clone the repository use the following
  5. To update existing projects on your machine to use the new address follow the instructions below
  6. Open your project’s git config file
  7. Change the url line to be

That’s it. Hope this helps anyone who is getting the error ‘Error: Permission denied (publickey)’.  There are other ways to do this but this way was pretty easy.

square

When loading partials into your one page AngularJS app occasionally you’ll need to load in JavaScript files that pertain only to those partials.  If you are only using AngularJS these added <script> tags won’t be rendered by Angular.  After Googling for some time I have found that the solution is to load JQuery into your index.html file just before AngularJS.  This fixes the problem.

partial.html

index.html

 

Problem solved. Hope this helps someone else who runs into the same problem I had.  If there is a better, more Angular way, of doing this let me know in the comments below.

replogo

Last time we looked at how to generate a very simple PDF using ReportLab and Django, ReportLab and Django – Part 1 – The Set Up and a Basic Example.  This time let’s make the PDF a little bit more interesting with some headers and footers.  ReportLab gives a pretty good amount of control when it comes to adding headers and footers to your PDF.  To start off let’s bring back the simple __init__ method we had from last time and add a header/footer method to it.

Now we have a method that will draw things onto every page of our header and footer.  Next we need to actually call that method.  Remember last time where we built the document using doc.build(elements) and ReportLab built a PDF using the flowable elements in the list?  Well this time we are just going to add some more parameters to let the build process know that it needs to also draw out the header and footer on every page.  Below is our print_users method from last time with the new parameters added to the build command shown in bold.

That’s it. Now our print_users method will print out a few lines on both the header and the footer of every page. You’ll notice there are two parameters, onFirstPage and onLaterPages. This allows you to have different headers and footers on the front page, which could be a title page, and on later pages that would require page numbers. Speaking of page numbers let’s get into that.

Page numbers on every page was a big challenge for me to find anything on. It was easy enough to put Page X on every page but what I really wanted was Page X of Y and that “of Y” portion was tricky. Luckily, I’ve figured it out and I will present it here for you now.

You can alter the way that the PDF is built using the canvasmaker parameter in the doc.build() process. These canvasmakers can help you inject new things on every page without using a header or footer. Below is the code that I found somewhere online to print the Page X of Y on every page down near the footer. I have placed this all within the printing.py file I created last time.

To use this we just add a new parameter to our doc.build() command. You can see it added below in bold.

There you have it. Now you have a great header and footer on every page, or every page after the first, and page numbers on all of them. How fantastic is that? Next time we will go a bit deeper into Tables and how to use them to line everything up really nicely. They will be usable wherever you want a flowable including your new headers and footers.

send-file-iconupload-files-icon-ts7tteez

As we continue to push forward on development our users have asked for a robust way of dragging and dropping images and linking them to a work order.  I was putting this off for a while because I thought it would be a pretty difficult task but after a few Google searches I found a great jQuery plugin from blueimp called jQuery-File-Upload.  Take a look at their demo and check out how awesome it is.  We didn’t need any of the fancy bells and whistles that came with the Basic Plus UI so this tutorial focuses on just the Basic but you can scale yours how you’d like.  The tutorial will also focus on getting it to work and save to a MySQL database through Django.

To start off let’s get the drag and drop area set up on our page.  To do that you’ll need to add these things to your HTML template, Django tags are added because that is what we are using to render the page.

image_upload.html

With the code above you should be ready to drag and drop files immediately, at least from the front end perspective. Now let’s set up the server side of things. Make sure you add your URL from the fileupload to your urls.py file. Then create a view for that URL that looks something like this.

views.py

The view handles all of the uploaded files one at a time and saves them to wherever we would like. In this case we are making a database entry that houses the file in an ImageField and we have a CharField for a name that we can change without altering the file path used in the ImageField. The JSON output is important as it will read the data sent back and can output our newly added images right to the screen.  That’s all there is to it.

Now there may be some circumstances where you would like to add more data to the request than just the images you have dragged over.  In our case we wanted the work order number to be attached as well and saved in the file name.  Using jQuery File Upload we can add data anytime something is dropped by adding a bind call to all fileuploadsubmit method calls.  I added the following lines of code in the HTML template just underneath the var url line.

This will add the workorder id to the POST request data on every call using the fileupload. Another thing to note is you can add a check to see if you really want to submit the uploads like I have done. If the workorder is not active do not upload any images. You can customize this any way you’d like since it is all just JavaScript and jQuery code.

Another problem I ran into was having multiple file upload elements on the same page. This was really frustrating and reading through their documentation they had a fix that worked pretty well with a couple of tweaks.

First, to make all of the drag and drop areas work independently of one another you change the fileupload elements from having an id of fileupload to a class of fileupload. This will need to change on the above code for bind as well. But you can’t simply change it on the $(‘#fileupload’).fileupload() becuase then it will upload once for every drag and drop element on the page. You instead have to take it a step back and add these lines.

This allows every fileupload classed element to take files individually. But this was annoying because users had to drop the files right on the button to make it work. As you can see there is a new parameter we’ve added called dropZone. This tells the fileupload what area of the page to watch for. If you want to make this area bigger or smaller that is totally up to you. I changed mine to dropZone: $(this).parents(".panel-body") to allow the user to drop images anywhere inside the panel body and they will start the upload.

I hope this was useful to someone. We are actually going in a different direction with our image uploads but we may need this in the future and I wanted to document how I got it all working with both jQuery File Upload and Django.

EDIT

Thanks to Tuan Nguyen for pointing out that in its default configuration each image is uploaded in its own request.  To join these into one request, open up the jQuery-File-Upload/js/jquery.fileupload.js file and search for the line singleUploadFiles: and set it to true.  You can also limit the number of files to be uploaded right below by changing limitMultiFileUploads:.

It’s been far too long since my last post.  A lot has happened and the holidays make things so busy but I’m back and ready to get back in to the swing of things.  To start off the new year I’m going to do a series of posts relating to ReportLab specifically with its application to Django.  I want to go over a few of the pitfalls I’ve experienced and how to get things set up.  ReportLab is notoriously tricky to get working and the documentation seems sparse to say the least.  I hope these posts help not only myself remeber how I did all of this but others who are struggling with the same problems.

ReportLab

The problem is simple, how to generate PDFs using Django.  In our on going rewrite of our system at work we are getting to PDF generation.  The old system uses wkhtmltopdf which is great if you just want something to render HTML out to a PDF and you don’t give much care to speed or formatting.  Wkhtmltopdf suffers from having to use HTML to format everything on the page.  This is limiting because it is difficult to add headers and footers, positioning objects on the page is tricky, and we have a bad problem with it printing blank pages due to random overflow of the elements.  We needed a better solution and we have found it with ReportLab.  ReportLab has some HTML support, and we’ll get to that in a later segment, but for the most part you are drawing elements, text, images, etc. directly onto a canvas that is then rendered as a PDF.  The speed increase for us was unbelievable.  The formatting and control that we get is unmatched.  Most importantly, the font support has given us cleaner paperwork than we ever thought possible.  I guess what I’m trying to say is ReportLab is great…once you get it to work.  ReportLab Documentation is broad but not specific enough for a lot of our concerns.  You’ll find yourself heading to Google for even the smallest of questions.  I hope to alleviate many of the common pitfalls with this series.

Getting Set Up

Start by installing reportlab into your virtual environment with pip install reportlab. The first thing to do is get a robust class set up for doing all of your printing.  I created a printing.py file that houses all of my basic ReportLab set up and functions.  Branching off from there I have my individual pages and paperwork that get combined and output to the user.  Let’s start with a simple printing.py file.

As you can see my class is very simple. I pass in a buffer to house my data that will be returned to the user, I’ll show this once we get set up, and a page size. ReportLab has a lot of built in page sizes but you can even set your own if you’d like by using your own width and height variables.

Simple Example – Printing Users

Now let’s add a simple method to print out all of the users of our application into a PDF and return it to the browser.  Inside of the MyPrint class I’m adding a method that will print the list of users.

The code is commented nicely but let’s walk through it.  We start by taking our buffer and putting it into what is a SimpleDocTemplate.  This is a ReportLab object that will generate a PDF with some standard values.  There are a ton of customizations you can make and even create your own document templates but for now let’s stick with the simple.  We set the margins for the page and the page size.  You can use inches or millimeters too if you’d like, ReportLab has a few built in measurement sizes that just need to be imported.

Next we have a list of elements.  This list will hold all of the ReportLab Flowables that will be generated on the PDF itself.  A Flowable is an object that has space and size on a PDF.  This can be a block of text, a table, an image, or you can even create a custom Flowable.  Whatever we put in this list will be generated in order on our PDF.

After that you’ll see some styles that we use.  These styles are going to be used for various elements like Paragraph styling.  After we get the style sheets I’ve added a new one that simply centers text and nothing more.

Now we come to the part where we are going to add elements to our document to be rendered.  I’ve used Django to grab my list of users and I loop through them one at a time to add their name to a Paragraph flowable.  A Paragraph flowable will take up the entire width of the area you give it.  In our case we have not confied the Paragraph to the inside of a table or to a column so each Paragraph will fill in the entire width of the document.

Once we have added all of our elements we simply build the doc.  This renders the PDF and places all PDF data into that buffer we gave the doc when we initiated it.  Close the buffer for good practice and return the contents of the PDF to the user.  Simple enough.  To run all of the code that I have shown I have a view in my views.py file that will run all of this and you can see how to access the printing.py class and method and output your PDF to your users.

That’s all there is to it. I hope this was enlightening on at least how to get started. Next time I will explain how to add headers and footers to your documents as well as a surprisingly tricky to find way of adding page numbers.

If you have anything you’d like to learn about for ReportLab and Django please leave a comment below.

ReportLab and Django – Part 2 – Headers and Footers with Page Numbers

Security-camera-sign

A couple of weeks ago it came out that there is a flaw in Django’s ImageField which could potentially allow for phising programs to be uploaded and grab cookies or do other malicious things.  While there will be no fix in Django directly you still need to take precautions on how you serve and receive files uploaded by your users.

Django has a page dedicated to fixing this exact issue.  Head over to their security guide and read up on the fixes.  They shouldn’t be too hard and shouldn’t affect any user experience.

Just wanted to post and let everyone know of the vulnerability.

git

I’ve found that a lot of things on Ubuntu are very easy to install but there are some trickery ones that if you don’t know the procedure can be a pain to get up and running on a new installation.  Here is a quick setup guide for getting SmartGit installed on a fresh Ubuntu installation.

  1. Install the Java Runtime Environment from here.  http://apt.ubuntu.com/p/openjdk-7-jre
  2. Install Git with the command sudo apt-get install git-core
  3. Install Mercurial with the command sudo apt-get install mercurial meld
  4. Download and install SmartGit per their instructions on their site

That’s all there is to it.  Hope this helps both myself and anyone else looking for a quick guide to get SmartGit up and running.

aws-logo

Amazon Web Services (AWS) has been revolutionary in getting servers and processing power to virtually anyone on the cheap.  You can host your web site there, process large sets of data, even back up your personal files to an Amazon S3 server.  Even if your site isn’t hosted directly on AWS you can still benefit by hosting your files from there.  A lot of sites do this to help off load the bandwidth costs onto Amazon because in many cases it is cheaper than the bandwidth costs for your site.  In this post I want to go over how to access these files stored on an S3 instance of AWS from Python using a plugin known as Boto.  You can read the Boto S3 commands at their website.  There are a lot of them and I am only going over the basics here.

Boto is a Python interface to Amazon Web Services and helps you interact with every aspect of AWS.  I’m just going to focus on the S3 portion since that is where we store files for clients.  We designed a way for clients to view and store files through our client portal.  It is sort of like Dropbox but not nearly as sophisticated.  We are able to drop files in their folder for them to view and they can even add in files for themselves.  Everything is saved back at Amazon and everyone is happy.  To accomplish all of this in our Django project we needed to interact with S3 as if it were a normal file system.  Since S3 is remote, connections need to be made, files need to be uploaded, and paths need to be verified.  One of the easiest things we did was create a FileUtils class that handled all of our Boto code and then simply created an instance of it and called our methods.  Let’s go through our code and feel free to change whatever you need to to make it work for you.  I’ve stripped out a lot of the more proprietary code that is uninteresting or unneeded in the course of this post.

There was a ton of code and this post was messy so I moved it all to a PasteBin link.

Let me explain everything that is happening here.  Because we are storing files specific to clients we need to keep them separated.  We gave them the ability to share documents through email addresses.  You’ll notice throughout that we make checks for FileSharesPermissions to verify the user has access to the file they are trying to access.  If you do not care about that and you just want to serve files, feel free to rip all of that out.

The FileUtils class is created by passing in a companyID.  Again, if you are not worried about separation of files just remove all of that.  What is important is a rootFolder, s3Connection, and s3Bucket.  We have opted to place these in the settings file as they will be used and changed as needed.  The rootFolder can be anything you want it to be.  S3 stores files using a key name that can be equated to a path to a file.  Inside of your bucket you will have keys organized by their name and sub categorized by their name just like a path to a file.  We have set up a rootFolder where we will be placing client files separated by their client ID.  Once again, if you don’t need to separate then drop all that code and stick the files wherever you’d like.

In the FileUtils class you’ll see a number of very basic file manager methods.  Save, create folder, delete, get file, get raw file, move, move file, get contents, and various helper methods to accomplish all of this.  Most of the methods should be self explanatory and are only complicated by the check for permission to access the file.

A few things of note.  Everytime you want to save a new file you need to create a new key using Boto.  You’ll see that under the save_file method that k = self.s3Bucket.new_key(path).  This is creating a new empty key, no data, at that name in the S3 instance.  It then sets its contents from a string, passed from a file upload, and is saved.  You do not need to worry about creating a folder or subfolder for the file.  S3 takes care of that using the name of the key.  For example, if I have a totally empty S3 instance and I try to save a file to Files/New Files/Some Other Folder/My PDF.pdf all I would need to do is set the name of the key that holds My PDF.pdf as that path name.  S3 will automatically read the slashes as folders and take care of the rest.  This makes things tricky for something like creating a new folder.  Something actually has to be in the folder to be considered by S3 to be a folder.  To get around this you’ll see under the create_folder method that we generate a key at that path and set the contents to nothing.  This creates the key, S3 displays it as a folder, but that folder is technically empty.  Now you can see it and save files to it whenever you’d like.

When you retrieve a file S3 generates a temproary URL.  Under the method get_file you’ll see that we return the url = k.generate_rul(120).  The 120 is how many seconds that temporary URL will be valid for.  This is great for security as it requires users to constantly regenerate the URL to ensure they still have access to it.  The get_raw_file on the other hand does not use the temporary URL and instead just gets the contents of the file and returns it to the user.  Both methods work and you should use the one that suits your needs.

This post might be really complicated and a bit hard to understand.  If anything needs to be clarified please feel free to comment below and I will be glad to rewrite and expound more on the code and workings of Boto.

grunt-logo

Developing for the web can be tedious sometimes.  Not only do you need to develop using multiple languages you need to make things as easy for yourself as you can when it comes to deployment.  One of the most basic things you can do to make your source files ready for production is to minify all of your JavaScript files.  There are a few programs out there that will do it for you but today I want to show you Grunt.

The great thing about Grunt is it will autocompile and minify your code even as you’re developing.  For example, in your templates you set the source for your scripts to the .min.js versions.  Normally once you make a change to the source you’d need to manually minify the file again before refreshing the page.  Grunt will do all of the minifying for you and using one of their plugins can watch your code for changes and automatically recompile when you save.

I’m going to go through a basic setup for Grunt.  There are many more advanced things you can do with Grunt but I’m just going to show you a basic setupd to recompile your LESS files and minify your JavaScript each time they change.  The first thing you’ll want to do is install Grunt.  To install it you’ll need NPM and Node.js.  If you already have those installed skip down to step 3.

  1. sudo apt-get install npm
  2. Download Node.js from http://nodejs.org/
    1. tar xvzf node...tar.gz
    2. cd node.../
    3. sudo ./configure
    4. sudo make
    5. sudo make install
  3. npm install -g grunt-cli
  4. Go to your project folder and run grunt init to create a basic package.json file.  You also need to create an empty Gruntfile.js.  These need to be at the root of your project.
  5. npm install grunt --save-dev
  6. npm install grunt-contrib-watch --save-dev
  7. npm install grunt-contrib-uglify --save-dev
  8. npm install grunt-contrib-less --save-dev

Step 3 installs the Grunt command-line interface.  Step 4 ensures you have the right files to install your Grunt setup.  Steps 5-8 install the plugins we are going to use in this guide.  Watch is the plugin to watch our files for changes and make something happen when they do.  Uglify is the minification plugin.  Less is what will compile our LESS files to CSS.  The --save-dev automatically puts the plugins into your package.json file.

Now that you have everything installed you need to set up your Gruntfile to do what you want Grunt to do.  Here is a simple example Gruntfile.js.

Let’s walk through that file.  At the top you see grunt.initConfig and in here we will put a dictionary with each key representing either a Grunt keyword or a Grunt plugin.  In the example above you can see there is pkg, watch, less, and uglify.  Pkg reads your package.json for information about your project.  watch, less, and uglify all contain information for the various plugins to know what to compile and where to output it.  I’ll quickly go through each plugin and how it is setup.

Watch contains a dictionary with named methods.  I’ve labeled them after the Django app it relates to in my project but you can name them whatever.  Inside of the labeled section are the keywords files and tasks.  Both are lists and they define what files to watch and what tasks to perform when one of those files have been changed.

Less is structured similarly to Watch but instead of triggering a task it is going to take source files and convert them to CSS files.  It takes one keyword argument with a list of dictionaries with various options.  To use those options you first need to set expand to true.  Cwd is the relative path name to your source files.  Src says where to look for and what type of file they are.  Here we are checking all directories and subdirectories found at cwd for .less files to convert.  Dest is where the CSS files will be outputted.  Ext says what the file extension should be.

Uglify looks almost exactly the same as the less section and that is because it is doing almost the exact same thing, taking source files and outputting files in a destination folder.  I created a sub-folder inside my js folder to house the source files and then send the minified versions to the main js folder.  This allowed me to keep most of my URLs in tact and just add .min.js to the filename.  At the top of uglify there is the options keyword with banner inside of it.  You can customize this to put a small bit of text at the top of your minified files.  Here we have the package name and when it was compiled.

Now that we have all of the plugins configured we need to activate them all.  Place all the names of the plugins you are using under the Plugins section.  After that you create tasks.  Every Gruntfile needs a default task that will be run when you use the command grunt.  I have mine set to do all the things inside of both uglify and less so it compiles all of my code in one command.  Next you’ll need to register the task names so watch knows what to run.  Once you’ve done that you are good to go.  Run the command grunt watch and then change any of your files.  You should see that Grunt automatically redid whatever section it was told to do.

Grunt is a great tool and I have only slightly scratched the surface of what it is capable of.  Please head over to their website to learn more about Grunt and the great things you can do with it.  If you have any tips or tricks with Grunt please comment below.