TIL from the Rocketship

TILSeptember 18, 2019by Alexander Spitsyn

Rewind Files Before Reading

Suppose you have a form with a file input and you need to forward that file to an external service through your Rails app. Params will look like this:


{ "file"=> #<ActionDispatch::Http::UploadedFile:0x00007fce9a8af140 @tempfile=#<Tempfile:/path/to/the/file.xlsx>, ... > }

The external service can not be reached for the first time sometimes, so you decided to set some retry count.


retry_count = 5

while retry_count > 0

    begin

        HTTPClient.new.post(url, params, headers)

    rescue ExternalServiceUnavailable => e

        retry_count -= 1

    end

end

The code above uses HTTPClient gem for making http requests and it will raise the following error in case there will be at least one retry:

ArgumentError: Illegal size value: #size returns 154139 but cannot read

The thing is that HTTPClient uses IO#read to read the file by chunks for making POST request. The lineno pointer reaches the end of the file after read, and you can not read bytes from the I/O stream anymore:


params['file'].tempfile.eof? # => true

To fix that, use IO#rewind – it positions IO to the beginning of input, resetting lineno to zero. Just add checking the file for eof before each request and rewind it if needed:


retry_count = 5

while retry_count > 0

    begin

        params['file'].tempfile.rewind if params['file'].tempfile.eof? # check eof

        HTTPClient.new.post(url, params, headers)

    rescue ExternalServiceUnavailable => e

        retry_count -= 1

    end

end

TILSeptember 17, 2019by Alexander Budchanov

Renaming keys in PostgreSQL JSON / Migration template for gem Recorder

In our project, we used gem Recorder. It saves changes in jsonb format.

Once we need to rename a column in model Contact, but we didn't want to lose logs.

I introduce you template of migration for this case.


class RenameOldColumnToNewColumn < ActiveRecord::Migration[5.2]

  def up

    rename_column :contacts, :old_column, :new_column



    execute <<~SQL

      UPDATE recorder_revisions SET data = jsonb_set(data #- '{changes,old_column}', '{changes,new_column}', data#>'{changes,old_column}') WHERE data#>'{changes}'?'old_column';

    SQL

  end



  def down

    rename_column :contacts, :new_column, :old_column



    execute <<~SQL

      UPDATE recorder_revisions SET data = jsonb_set(data #- '{changes,new_column}', '{changes,old_column}', data#>'{changes,new_column}') WHERE data#>'{changes}'?'new_column';

    SQL

  end

end

You can simple replace contacts, new_column and old_column to your table name and columns names.

TILSeptember 18, 2019by Andrey Morozov

Strip Whitespace from Heredocs in Ruby/Rails

UPDATE!

Simple way - it's use:


class Template

  def self.base

    <<~TEXT

      Lorem Ipsum is simply dummy

      Lorem Ipsum is simply dummy

    TEXT

  end

end



> Template.base

=> "Lorem Ipsum is simply dummy\nLorem Ipsum is simply dummy\n"

I forgot about <<~, and that's why I got this TIL bike.


You use heredoc in Ruby/Rails app?

Then you're familiar with the problem..


class Template

  def self.base

    <<-TEXT

      Lorem Ipsum is simply dummy

      Lorem Ipsum is simply dummy

      Lorem Ipsum is simply dummy

    TEXT

  end

end



> Template.base

=> "      Lorem Ipsum is simply dummy\n      Lorem Ipsum is simply dummy\n      Lorem Ipsum is simply dummy\n"

hmm...

I don't want to have so many gaps in the text.

Maybe use them:


class Template

  def self.base

    <<-TEXT

Lorem Ipsum is simply dummy

Lorem Ipsum is simply dummy

    TEXT

  end

end



> Template.base

=> "Lorem Ipsum is simply dummy\nLorem Ipsum is simply dummy\n"

It's good, but the code looks like 💩.

This problem can be solved by using activesupport #strip_heredoc:


class Template

  def self.base

    <<-TEXT.strip_heredoc

      Lorem Ipsum is simply dummy

      Lorem Ipsum is simply dummy

    TEXT

  end

end



> Template.base

=> "Lorem Ipsum is simply dummy\nLorem Ipsum is simply dummy\n"

If you don't have active_support, use refine String with #strip_heredoc which will give similar result:


module Utils 

  refine String do 

    def strip_heredoc

      gsub(/^#{scan(/^[ \t]*(?=\S)/).min}/, "".freeze)

    end

  end

end



class Template

  using Utils

  def self.base

    <<-TEXT.strip_heredoc

      Lorem Ipsum is simply dummy

      Lorem Ipsum is simply dummy

    TEXT

  end

end



> Template.base 

=> "Lorem Ipsum is simply dummy\nLorem Ipsum is simply dummy\n"

TILSeptember 17, 2019by Alexander Budchanov

Rails 5.2 changes in callbacks

In version 5.1 you may see deprecation warnings in after_save callbacks (related to changes in ActiveRecord::Dirty module).

But since 5.2 these changes were applied.

For examples, I will use Rails 4.2.11 and Rails 5.2.3 and model User with email attribute. Let's do:


u = User.new(email: 'old@domain.com')

u.save

u.email = 'new@domain.com'

u.save

and look at after_save callback in time of last save.

1. attribute_changed?

Rails 4


> email_changed?

=> true

Rails 5.2


> email_changed?

=> false

but you can use saved_changes?


> saved_change_to_email?

=> true

2. changed?

Rails 4


> changed?

=> true

Rails 5.2


> changed?

=> false

but you can use saved_changes?


> saved_changes?

=> true

3. changes

Rails 4


> changes

=> {"email"=>["old@domain.com", "new@domain.com"]}

Rails 5.2


> changes

=> {}

but you can use saved_changes


> saved_changes

=> {"email"=>["old@domain.com", "new@domain.com"]}

4. previous_changes

Rails 4


> previous_changes

=> {"email"=>[nil, "old@domain.com"]}

Rails 5.2

Now, this method returns the changes that were just saved (like saved_changes).


> previous_changes

=> {"email"=>["old@domain.com", "new@domain.com"]}

this method has no replacement.

TILSeptember 16, 2019by Andrey Morozov

Simple way to get all values from hash

In the ruby from the box, we can't find all the hash values if it's nested.

I suggest an easy way to find all the values using recursion.

Example hash:


hash = {

  a: 2,

  b: { c: 3, d: 4, e: {f: 5}}

}


> hash.values

=> [2, {:c=>3, :d=>4, :e=>{:f=>5}}]

That's not an option, we need all the values.


def deep_values(array = [], object:)

  object.each do |_key, value|

    if value.is_a?(Hash)

      deep_values(array, object: value)

    else

      array << value

    end

  end

  array

end



> deep_values(object: hash)

=> [2, 3, 4, 5]

If you run the benchmark with this data, we get the following data:


>  puts Benchmark.measure { 100_000.times { hash.values } }

=> 0.028920   0.002643   0.031563 (  0.032759)



>  puts Benchmark.measure { 100_000.times { deep_values(object: hash ) } }

=> 0.140439   0.003318   0.143757 (  0.146637)

TILSeptember 13, 2019by Igor Alexandrov

How to parse CSV with double quote (") character in Crystal

We use microservice written in Crystal to parse large CSV files (about 1.5Gb). Some rows in these files may contain no closed " characters:


,Y,FEDERAL NATIONAL MORTGAGE ASSOCIATION "F,,

With Crystal default CSV parse settings this row and everything after it won't be parsed correctly because DEFAULT_QUOTE_CHAR constant is equal to ". Of couse you can override quote_char param in CSV contstructor with something that cannot be found in your document.

From my point of view the best is to use zero byte which is '\u0000' in Crystal.


csv = CSV.new(file, headers: true, strip: true, quote_char: '\u0000')

while csv.next

  # ... 

end

Hack!

TILSeptember 13, 2019by Dmitry Voronov

How to use Rails translations for Reform attributes

If you use Reform for the form objects and want the translations in your Rails application to work as with ActiveRecord objects, then you can add ActiveModel::Translation module to the form class or base class and specify the method with the key to translations.


class BaseForm < Reform::Form

  extend ActiveModel::Translation



  def self.i18n_scope

    :forms

  end

end


class UserForm < BaseForm

  feature Coercion



  property :login

  property :change_password, virtual: true, type: Types::Form::Bool

  property :password

  property :password_confirmation



  validation do

    required(:login).filled

  end



  validation if: ->(_) { change_password } do

    required(:password).confirmation

  end

end

Set translations for form fields.

You can move them to a new file config/locales/forms.yml.


en:

  forms:

    attributes:

      user_form:

        login: Enter login

        change_password: Change password?

        password: Enter password

        password_confirmation: Confirm password

Now, translations will be used by the simple form or you can call them manually by UserForm.human_attribute_name.

TILSeptember 05, 2019by Alexander Ivlev

Redux async actions. Tracking loading and errors with React hooks.

If you use redux and async actions, then you probably had to deal with a situation where you need to monitor the loading and error status of this action. With the advent of hooks, it became possible to conveniently transfer this logic to one block and use it everywhere.


import { useState, useCallback } from 'react';

import { useDispatch } from 'react-redux';



function useAsyncAction(action, dependeces = []) {

  const dispatch = useDispatch();

  const [loading, setLoading] = useState(false);

  const [isError, setIsError] = useState(false);



  const asyncAction = useCallback(

    (...args) => {

      async function callback() {

        setLoading(true);

        try {

          const res = await dispatch(action(...args));

          setIsError(false);

          setLoading(false);

          return res;

        } catch (e) {

          setLoading(false);

          setIsError(true);

          return e;

        }

      }

      callback();

    },

    [action, ...dependeces],

  );



  return [asyncAction, loading, isError];

}

Now you can use this hook in your functional component.


// …

  const [load, loading, isError] = useAsyncAction(() => loadActivityRequest(applicationId), [applicationId]);

// …

  load();

TILSeptember 04, 2019by Vitaly Platonov

Postgres. Search Array type columns

Say we have a table with a column of an array type. At some point, we want to be able to select records with a specific value(s) which the array column may have.

Here are three ways to do different kinds of searches.

1) Use ANY operator when searching with one value:


SELECT * FROM mytable WHERE 'first_type' = ANY(types_column);

2) Go with the "contains" operator (“@>”) when you look for a specific set of values (the order of values doesn’t matter):


SELECT * FROM mytable WHERE types_column @> '{"first_type", "second_type"}';

The values “first_type” and “second_type" must be in the types_column column for a record to be selected.

3) Whenever you need to search any values that a column may have - use the "overlap" operator (“&&”)


SELECT * FROM mytable WHERE types_column && '{"first_type", "second_type"}';

One of the values “first_type” or “second_type" must be in the types_column column for a record to be selected.

TILSeptember 03, 2019by Andrey Morozov

Git switch command

A switch command has been added in the new version of git

Let's look at examples:


# switched to <branch>

git switch <branch>



# creates a new <branch>

git switch -c <branch>



# switched to commit 

git switch -d <commit> 



# creates and switches to branch from remote. 

# need to use if branch exists in multiple remotes 

git switch -c <branch> --track <remote>/<branch> 



# switch to a branch even if the index or working tree is different from HEAD

# this is used to throw away local changes

git switch --discard-changes <branch>



# alias for  --discard-changes

git switch -f <branch> 



# switch back to the previous branch before we switched

git switch - 

Command available on version Git 2.23.0 or higher

TILSeptember 02, 2019by Alexander Spitsyn

Determining class of an object with case equality operator (===)

Case equality operator (or triple equals, ===) in Ruby returns true if the passed class is in the ancestors list of the passed object's class:


1.class.ancestors # [Integer, Numeric, Object, ...]

Numeric === 1 # true

Object === 1 # true

So it can be used for determining object's class:


String === 'abc' # true

'abc'.class #=> String

In cases above the case equality operator works like #kind_of? (or #is_a?):


1.kind_of?(Integer) # true

1.is_a?(Numeric) # true

The classes above has different implementations of === operator, that's why the results of comparison are different:


String.===('abc') # the same as String === 'abc'

Also it means that order of the arguments is important:


1 === Integer # false

TILAugust 30, 2019by Alexander Spitsyn

Testing external API integration with VCR gem

Suppose your application connects to an external service via API and you have a wrapper for this API that handles and parses response. The VCR gem gives you ability to store parsed response in a special format (cassetes). VCR makes a real request to the API for the first test run and writes it's response to the cassete for next test runnings.

First you need to set VCR configuration:


VCR.configure do |config|

  config.cassette_library_dir = "spec/vcr_cassettes"

  config.hook_into :webmock

end

And then write specs like the following:


RSpec.describe ExternalService do

  describe '#new_order' do

    let(:params) { { foo: 'bar' } }



    it 'creates new order' do

      VCR.use_cassette("external_service") do

        result = subject.new_order(params) # makes HTTP POST to an external service



        expect(result.successful?).to be_truthy

        expect(result.data).to have_key('order_id')

      end

    end

  end

end

See more details on the official page of the gem.

TILAugust 28, 2019by Ilia Kriachkov

Ruby double splat (**) operator cheatsheet

The operator ** is useful as an options hash.


def one_method(**options);end

This form is completely similar to the following:


def another_method(options = {});end

In addition, you can strictly define the set of required keys for the method.


def one_strict_method(first_name:, last_name: , **options)

  puts "options: #{options}"

  greeting = "Hello #{first_name} #{last_name}"

  puts options[:upcase] ? greeting.upcase : greeting

end


pry(main)> one_strict_method(upcase: true)

ArgumentError: missing keywords: first_name, last_name



pry(main)> one_strict_method(first_name: 'John', last_name: 'Doe', upcase: true)

options: {:upcase=>true}

HELLO JOHN DOE

=> nil

Another advantage of double splat literal is that it works like #merge for Ruby Hash


class Contact::ShowRepresenter #:nodoc:

  def call(contact)

    {

      contact: {

        **base_info(contact),

        **legal_info(contact),

        # You can add something more complex here.

        # **GeoLocaionRepresenter.new.(contact)

      }

    }

  end



  private



  def base_info(contact)

    {

      id: contact.id,

      first_name: contact.first_name,

      last_name: contact.last_name,

      email: contact.email,

      phone: contact.phone

    }

  end



  def legal_info(contact)

    {

      legal_name: contact.legal_name,

      legal_type: contact.legal_type,

      mailing_address: contact.mailing_address

    }

  end

end


pry(main)>Contact::ShowRepresenter.new.(Contact.last)

=> {

  :contact=>{

    :id=>51986,

    :first_name=>"Ilia",

    :last_name=>"Kriachkov",

    :email=>"ilia.kriachkov@jetrockets.ru",

    :phone=>"+79000000000",

    :legal_name=>"JetRockets",

    :legal_type=>"LLC",

    :mailing_address=>"15 Let Oktyabrya Street, #10b, Tver, Russian Federation 170008"

  }

}

In conclusion, I want to demonstrate some benchmark results.

As you can see, the ** operator is a bit faster than Hash#merge.


require 'benchmark'

n = 50_000

Benchmark.bm(2) do |x|

  x.report('merge:             ') { n.times { merge } }

  x.report('double_splat_merge:') { n.times { double_splat_merge } }

end



def merge

  hash = { a: 'a' }

  { b: 'b' }.merge(hash)

end



def double_splat_merge

  hash = { a: 'a' }

  { b: 'b', **hash }

end


                     user      system      total        real

merge:               0.109247   0.088652   0.197899 (  0.204470)

double_splat_merge:  0.079480   0.003590   0.083070 (  0.083642)

TILAugust 26, 2019by Dmitry Voronov

How to store large JSON in PostgreSQL with Rails Attributes API

If you store large objects in the database (such as JSON), for example, data for big reports, then this can take up a lot of space. To reduce the size of data, you can compress and store in binary form.

PostgreSQL has a bytea field type for storing such data. You can add bytea column in Rails using migration


add_column :reports, :data, :binary

For binary field operations, you can use the Rails Attributes API and add a new BinaryHash data type


# app/types/binary_hash.rb



class BinaryHash < ActiveRecord::Type::Binary

  def serialize(value)

    super value_to_binary(value.to_json)

  end



  def deserialize(value)

    super case value

          when NilClass

            {}

          when ActiveModel::Type::Binary::Data

            value_to_hash(value.to_s)

          else

            value_to_hash(PG::Connection.unescape_bytea(value))

          end

  end



  private



  def value_to_hash(value)

    JSON.parse(

      ActiveSupport::Gzip.decompress(value),

      symbolize_names: true

    ) || {}

  end



  def value_to_binary(value)

    ActiveSupport::Gzip.compress(value)

  end

end

Register new type in initializers


# config/initializers/types.rb



ActiveRecord::Type.register(:binary_hash, BinaryHash)

And add to binary type attribute in model


# app/models/snapshot.rb



class Reports < ApplicationRecord

  attribute :data, :binary_hash

end

Tests show that data size is reduced by almost 3 times


Run time with 100000 width JSON

                           user     system      total        real

Compress JSON          0.008671   0.001535   0.010206 (  0.010885)

Decompress JSON        0.001357   0.000095   0.001452 (  0.001509)



json size       95450 bytes

binary size   33868 bytes

~ 2.82 times compression

TILAugust 26, 2019by Dmitry Voronov

A simple way to distribute jobs in Sidekiq queues

This option implies that jobs of one context are executed sequentially in one queue, and jobs of different contexts in parallel in different queues.

Let's look at the following example.

There are investment funds for which we want to make time-consuming reporting calculations. Jobs for calculation within the same fund are carried out sequentially so that there are no errors in the calculations, jobs of different funds are performed in parallel.

We automate the distribution of jobs in queues so as not to specify a queue manually.

Specify the queues in the sidekiq.yml configuration file:


:queues:

  - fund_processor_0

  - fund_processor_1

  - fund_processor_2

  - fund_processor_3

  - fund_processor_4

It is important that the queues are numbered from 0.

Now, when starting the worker, we indicate in which queue we will set the job depending on the fund. To do this, we use the operation of obtaining the remainder from dividing the fund ID and the count of queues. So we get the queue number.


# 5 queues

# fund ID % count of queues = queue number

# 1 % 5 => 1

# 2 % 5 => 2

# 3 % 5 => 3

# 4 % 5 => 4

# 5 % 5 => 0

def queue_name(fund_id)

  queue_number = fund_id % 5

  "fund_processor_#{queue_number}"

end

Start the worker, indicating to him the received queue name.

This can be done using the Sidekiq API


Sidekiq::Client.push(

  'queue' => queue_name(fund_id),

  'class' => Fund::ReportCalculator,

  'args' => [fund_id]

)

TILAugust 23, 2019by Eugene Komissarov

Rubyists life made easier with composition operators.

If you write Ruby code and wandered into FP world you might just started writing those little tiny methods inside your classes/modules. And that was awesome to write code like this:


class Import

  # Some code goes here...



  def find_record(row)

    [ Msa.find_or_initialize_by(region_name: row[:region_name], city: row[:city], state: row[:state], metro: row[:metro] ), row ]

  end



  # record is one of:

  # Object - when record was found

  # false - when record was not found

  def update_record(record, attributes)

    record.attributes = attributes

    record

  end



  # record is one of:

  # false

  # ZipRegion

  def validate_record(record)

    case record

    when false

      [:error, nil]

    else

      validate_record!(record)

    end

  end



  # record is ZipRegion object

  def validate_record!(record)

    if record.valid?

      [:ok, record]

    else

      error(record.id, record.errors.messages)

      [:error, record]

    end

  end



  def persist_record!(validation, record)

    case validation

    when :ok

      record.save

    when :error

      false

    end

  end

end

Yeah, I know there is YARD, and argument types are somewhat weird but at the time of coding, I was fascinated with Gregor Kiczales's HTDP courses (that was a ton of fun, sincerely recommend for every adventurous soul).

And next comes dreadful composition:


def process(row, index)

    return if header_row?(row)



    success(row[:region_name], persist_record!(*validate_record(update_record(*find_record(parse_row(row))))))

  end

The pipeline is quite short but already hard to read. Luckily, in Ruby 2.6 we now have 2 composition operators: Proc#>> and its reverse sibling Proc#<<.

And, with a bit of refactoring composition method becomes:


def process(row, index)

    return if header_row?(row)



    composition = method(:parse_row) >>

                             method(:find_record) >>

                             method(:update_record) >>

                             method(:validate_record) >>

                             method(:persist_record!) >>

                             method(:success).curry[row[:region_name]]



    composition.call(row)

Much nicier, don't you think? Ruby just became one step closer to FP-friendly languages family, let's hope there'll be more!

TILAugust 02, 2019by Maxim Romanov

How to blur a screen in React Navigation

Screens overlap each other in stackNavigator. React Navigation provides us not only with changing the background color of these screens, but also controlling their transparency.

To make the screen background blur, we first need to make the screens transparent.


import { createStackNavigator } from 'react-navigation';



export default createStackNavigator(

  {

    HomeStack,

    BlurModal,

  },

  {

    ...NAVIGATOR_CONFIG,

    mode: 'modal',

    cardStyle: { opacity: 1 },

    transparentCard: true,

  },

);

And then use blurView as background.


import React from 'react';

import { BlurView } from '@react-native-community/blur';

import Styled from 'styled-components';



function BlurModal() {

  return (

    <BlurWrapper blurType="light" blurAmount={20}>

      <Text>Modal with blur background</Text>

    </BlurWrapper>

  );

}



const BlurWrapper = Styled(BlurView)`

  position: absolute;

  top: 0;

  left: 0;

  bottom: 0;

  right: 0;

`;

TILJuly 31, 2019by Igor Alexandrov

Migrate tags in Rails to PostgreSQL array from ActsAsTaggableOn

ActsAsTaggableOn is a swiss army knife solution if you need to add tags to your ActiveRecord model.

Just by adding one gem to your Gemfile and acts_as_taggable to the model you get everything you need: adding tags, searching for a model by tag, getting top tags, etc. However, sometimes you don't need all these.

In our project, we used acts_as_taggable to store tags for Note model. Then we displayed a list of notes on several pages with assigned tags and had autocompleted input for tags on Note form. Everything worked well, but since we use PostgreSQL, I decided to store tags as an array in Note model.

First of all, I added tags Array<String> column to Note, after this migrated actsastaggable tags to notes table with migration.


class MigrateNoteTags < ActiveRecord::Migration[5.2]

  def change

    execute <<-SQL

    UPDATE notes 

    SET tags = grouped_taggings.tags_array 

    FROM

      (

      SELECT

        taggings.taggable_id,

        ARRAY_AGG ( tags.NAME ) tags_array 

      FROM

        taggings

        LEFT JOIN tags ON taggings.tag_id = tags.ID 

      WHERE

        taggable_type = 'Note' 

      GROUP BY

        taggings.taggable_id 

      ) AS grouped_taggings 

    WHERE

      notes.ID = grouped_taggings.taggable_id

    SQL

  end

end

To have backward compatibility, I added Note#tag_list method:


def tag_list

  tags.join(', ')

end

The last thing is to add the ability to search for tags. Since there about 500k records in the Notes table, I decided to create an SQL view:


CREATE OR REPLACE VIEW note_tags AS



SELECT UNNEST

  ( tags ) AS name,

  COUNT ( * ) AS taggings_count 

FROM

  notes 

GROUP BY

  name 

That's it! It takes from 100ms to 150ms to search for tags in this view, which is fine for me.

If you have more significant data sets, then the best would be to create tags table and add triggers to notes table that will update tags on INSERT/UPDATE/DELETE.

TILJuly 27, 2019by Alexander Ivlev

Request Api Adapter

When developing client applications, it is often necessary to send requests to the server.


// ...

client({ url: "/users.json", method: "GET" }).then(...)



We can make our lives a little easier. A convenient abstraction is apiAdapter


// apiAdapter.js

function getUsers() {

  return { url: "/users.json", method: "GET" };

}



const apiAdapter = createAdapter(client, {}, {

  getUsers,  

})

By defining a request in one place, you can now simply call the adapter method you want.


import apiAdapter from './apiAdapter'



apiAdapter.getUsers().then(...)

It is also a useful option to specify basic settings for all requests, as well as handling errors and successful requests.


const apiAdapter = createAdapter(

  client,

  { withCredentials: true },

  {

    getUsers,  

  },

  successHandler,

  errorHandler, 

)

Github

Live example

TILJuly 22, 2019by Alexey Belousov

Networking and thermal conditions debugging in Xcode 11

Xcode 11 introduces tool for networking and thermal conditions debugging. This feature requires device running iOS 13. Previously network conditioning feature was available within Additional Tools for Xcode.

See also