Advanced record generation

See the previous tutorial.

A typical case in system evolution is the growing complexity of entities into more detailed records. For example a system that maintains grouping can be expanding in more interlinked groups in the new system. This might generate more (linked) entities. data-migrator is build for these kind of questions.

One to many records

Linked to every model is a manager, which contains all logic to parse and emit records. By default this is a simple manager that scans one record per row. In those cases the system has to generate many, this is easily achieved by adding a dedicated manager:

from data_migrator import models

class ResultManager(models.Manager):
  def transform(self, row, previous, model):
    res = [model().scan(row), model().scan(row), model().scan(row)]
    res[1].a += 'some postfix'
    res[2].a += 'alternative postfix'
    return res

class Result(models.Model):
  id = models.IntField(pos=0) # keep id
  a  = models.StringField(pos=1)
  b  = models.StringField(pos=2, nullable=None)

  class Meta:
    manager = ResultManager

This example is not really helpful, it just generates 3 records instead of one.

Multi models generation

The manager setup is one of the big reasons to use this package instead of simple scripts. In our model migrations we came across steps in which we had to generate second models (for example permissions or links between models).

from data_migrator import models

class PermissionManager(models.Manager):
  def transform(self, row, previous, model):
    defaults = {
      'model_id': previous[0][0].a, # due to order permission will get a result
      'user': row[0]
    }
    res = [model(right='see_result', **defaults)]
    return res

class Permission(models.Model):
  id       = models.IntField()
  right    = models.StringField()
  model_id = models.StringField()
  user     = models.StringField()

  class Meta:
    manager = PermissionManager

class Result(models.Model):
  id = models.IntField(pos=0) # keep id
  a  = models.StringField(pos=1)
  b  = models.StringField(pos=2, nullable=None)


if __name__ == "__main__":
  transform.Transformer(models=[Result, Permission], emitter=MySQLEmitter).process()

The order in the transformer list determine the order of object creation, but this is an example on how one model can be used in the generation of the second one.