Igor Alexandrov

LongreadsMay 19, 2020by Igor Alexandrov

Scaling Rails: Docker & AWS Beanstalk

Scaling Rails project still can be quite an interesting task.

We always want deployment to be as easy for developers as possible. On the other hand we want to pay less for the infrastructure, but be ready for unexpected significant increase of RPM.

In this article we show how to automate deploy of Rails application with Sidekiq and AnyCable to AWS Elastic Beanstalk using GitLab.

Read more
TILFebruary 19, 2020by Igor Alexandrov

Migrating user passwords from Django to Ruby

One of our clients asked us to migrate his existing Django application to Ruby. A part of this process was a migration of an existing users database.

Of course we had to find a way to use existing crypto passwords from Django and not asking our users to reset them

Passwords in Django are stored as a string that consists of four parts splitted with a dollar sign.


<algorithm>$<iterations>$<salt>$<hash>

By default, Django uses the PBKDF2 algorithm with a SHA256 hash. I found a rather outdated Ruby gem that implements a Password-Based Key-Derivation Function and gave it a try.


def migrate_django_password_seamlessly(user, password)

  alg, iteration, salt, django_password = user.crypted_django_password.split('$')

  attempt = Base64.encode64(

    PBKDF2.new(password: password, salt: salt, iterations: 36000).bin_string

  ).strip



  # check if hash of user provided password equals to the password in a database

  if attempt == django_password

    user.update(password: password, password_confirmation: password)

  end

end

The method above is a part of user sign in service and called only if crypted_django_passwordcolumn from users table is not null.

LongreadsDecember 20, 2019by Igor Alexandrov

Two edge cases in PostgreSQL full-text search

We widely use PostgreSQL full-text search in our projects. It is fast, reliable, and doesn't add any additional technical complexity. But sometimes it may not work as you expect it too.

Read more
TILNovember 26, 2019by Igor Alexandrov

How to add HTTP Basic auth to Amber application

As you already may know, one of our projects have a Crystal application in production. It is created with Amber framework and works just perfect.

The only thing that I don't personally like in Amber is not clear and sometimes outdated documentation, after about 11 years with Rails I still think that Rails Guides are the number one developer documentation in the world.

I had a task to add HTTP Basic auth to a couple of URLs in Amber application, and after studying documentation found that Amber doesn't provide necessary Pipe out of the box. Ok, next place to search for the answer was Gitter and after about an hour Dru Jensen helped me with a code example.

Amber uses internally HTTP::Handler for Pipes as well as Kemal does for Middlewares, so we can easily use code from Basic Auth for Kemal.


# src/pipes/http_basic_auth_pipe.cr



require "crypto/subtle"



class HTTPBasicAuthPipe

  include HTTP::Handler

  BASIC = "Basic"

  AUTH = "Authorization"

  AUTH_MESSAGE = "Could not verify your access level for that URL.\nYou have to login with proper credentials"

  HEADER_LOGIN_REQUIRED = "Basic realm=\"Login Required\""



  property credentials : Credentials?



  def initialize(@credentials : Credentials)

  end



  def initialize(username : String, password : String)

    initialize({ username => password })

  end



  def initialize(hash : Hash(String, String))

    initialize(Credentials.new(hash))

  end



  def initialize

    if ENV["HTTP_BASIC_USERNAME"]? && ENV["HTTP_BASIC_PASSWORD"]?

      initialize(ENV["HTTP_BASIC_USERNAME"], ENV["HTTP_BASIC_PASSWORD"])

    end

  end



  def call(context)

    if credentials

      if context.request.headers[AUTH]?

        if value = context.request.headers[AUTH]

          if value.size > 0 && value.starts_with?(BASIC)

            return call_next(context) if authorized?(value)

          end

        end

      end

      headers = HTTP::Headers.new

      context.response.status_code = 401

      context.response.headers["WWW-Authenticate"] = HEADER_LOGIN_REQUIRED

      context.response.print AUTH_MESSAGE

    else

      call_next(context)

    end

  end



  private def authorized?(value)

    username, password = Base64.decode_string(value[BASIC.size + 1..-1]).split(":")

    credentials.not_nil!.authorize?(username, password)

  end



  class Credentials

    def initialize(@entries : Hash(String, String) = Hash(String, String).new)

    end



    def authorize?(username : String, given_password : String) : String?

      test_password = find_password(username, given_password)

      if Crypto::Subtle.constant_time_compare(test_password, given_password)

        username

      else

        nil

      end

    end



    private def find_password(username, given_password)

      # return a password that cannot possibly be correct if the username is wrong

      pw = "not #{given_password}"



      # iterate through each possibility to not leak info about valid usernames

      @entries.each do |(user, password)|

        if Crypto::Subtle.constant_time_compare(user, username)

          pw = password

        end

      end



      pw

    end

  end

end

And in routes.cr:


  # ...

  pipeline :api do

    # ...

    plug HTTPBasicAuthPipe.new

  end

  # ...

TILOctober 10, 2019by Igor Alexandrov

Double splat arguments in Crystal

In Crystal, as well as in Ruby you can use double splat arguments. Unfortunately they behave a bit different.


def foo(**options)

  baz(**options, a: 1)

end



def baz(**options)

  puts options

end



foo(b: 2, a: 3) # {:b=>2, :a=>1} 

This code in Ruby works as it should. If we try the same in Crystal (https://play.crystal-lang.org/#/r/7r0l), we got an error:


error in line 2

Error: duplicate key: a

This happens because **options is a NamedTuple and it cannot have duplicate keys. I found that using NamedTuple#merge can be a workaround (https://play.crystal-lang.org/#/r/7s1c):


def foo(**options)

  baz(**options.merge(a: 1))

end



def baz(**options)

  puts options

end



foo(b: 2, a: 3) # {:b=>2, :a=>1} 

Hack!

TILSeptember 13, 2019by Igor Alexandrov

How to parse CSV with double quote (") character in Crystal

We use microservice written in Crystal to parse large CSV files (about 1.5Gb). Some rows in these files may contain no closed " characters:


,Y,FEDERAL NATIONAL MORTGAGE ASSOCIATION "F,,

With Crystal default CSV parse settings this row and everything after it won't be parsed correctly because DEFAULT_QUOTE_CHAR constant is equal to ". Of couse you can override quote_char param in CSV contstructor with something that cannot be found in your document.

From my point of view the best is to use zero byte which is '\u0000' in Crystal.


csv = CSV.new(file, headers: true, strip: true, quote_char: '\u0000')

while csv.next

  # ... 

end

Hack!

TILJuly 31, 2019by Igor Alexandrov

Migrate tags in Rails to PostgreSQL array from ActsAsTaggableOn

ActsAsTaggableOn is a swiss army knife solution if you need to add tags to your ActiveRecord model.

Just by adding one gem to your Gemfile and acts_as_taggable to the model you get everything you need: adding tags, searching for a model by tag, getting top tags, etc. However, sometimes you don't need all these.

In our project, we used acts_as_taggable to store tags for Note model. Then we displayed a list of notes on several pages with assigned tags and had autocompleted input for tags on Note form. Everything worked well, but since we use PostgreSQL, I decided to store tags as an array in Note model.

First of all, I added tags Array<String> column to Note, after this migrated actsastaggable tags to notes table with migration.


class MigrateNoteTags < ActiveRecord::Migration[5.2]

  def change

    execute <<-SQL

    UPDATE notes 

    SET tags = grouped_taggings.tags_array 

    FROM

      (

      SELECT

        taggings.taggable_id,

        ARRAY_AGG ( tags.NAME ) tags_array 

      FROM

        taggings

        LEFT JOIN tags ON taggings.tag_id = tags.ID 

      WHERE

        taggable_type = 'Note' 

      GROUP BY

        taggings.taggable_id 

      ) AS grouped_taggings 

    WHERE

      notes.ID = grouped_taggings.taggable_id

    SQL

  end

end

To have backward compatibility, I added Note#tag_list method:


def tag_list

  tags.join(', ')

end

The last thing is to add the ability to search for tags. Since there about 500k records in the Notes table, I decided to create an SQL view:


CREATE OR REPLACE VIEW note_tags AS



SELECT UNNEST

  ( tags ) AS name,

  COUNT ( * ) AS taggings_count 

FROM

  notes 

GROUP BY

  name 

That's it! It takes from 100ms to 150ms to search for tags in this view, which is fine for me.

If you have more significant data sets, then the best would be to create tags table and add triggers to notes table that will update tags on INSERT/UPDATE/DELETE.