MongoDB Schema Design – Many small documents or fewer large documents?

Source : Stackoverflow

Modeling One-to-Few

An example of “one-to-few” might be the addresses for a person. This is a good use case for embedding – you’d put the addresses in an array inside of your Person object.

One-to-Many

An example of “one-to-many” might be parts for a product in a replacement parts ordering system. Each product may have up to several hundred replacement parts, but never more than a couple thousand or so. This is a good use case for referencing – you’d put the ObjectIDs of the parts in an array in product document.

One-to-Squillions

An example of “one-to-squillions” might be an event logging system that collects log messages for different machines. Any given host could generate enough messages to overflow the 16 MB document size, even if all you stored in the array was the ObjectID. This is the classic use case for “parent-referencing” – you’d have a document for the host, and then store the ObjectID of the host in the documents for the log messages.

How should I implement this schema in MongoDB?

User:
   username_name: string

Campaign:
   title: string
   description: string
   link: string

UserCampaign:
   user_id: integer
   camp_id: integer

Click:
   os: text
   referer: text
   camp_id: integer
   user_id: integer

(Source)

MongoDB relationships: embed or reference?

(source)

  • Put as much in as possible

    The joy of a Document database is that it eliminates lots of Joins. Your first instinct should be to place as much in a single document as you can. Because MongoDB documents have structure, and because you can efficiently query within that structure (this means that you can take the part of the document that you need, so document size shouldn’t worry you much) there is no immediate need to normalize data like you would in SQL. In particular any data that is not useful apart from its parent document should be part of the same document.

  • Separate data that can be referred to from multiple places into its own collection.

    This is not so much a “storage space” issue as it is a “data consistency” issue. If many records will refer to the same data it is more efficient and less error prone to update a single record and keep references to it in other places.

  • Document size considerations

    MongoDB imposes a 4MB (16MB with 1.8) size limit on a single document. In a world of GB of data this sounds small, but it is also 30 thousand tweets or 250 typical Stack Overflow answers or 20 flicker photos. On the other hand, this is far more information than one might want to present at one time on a typical web page. First consider what will make your queries easier. In many cases concern about document sizes will be premature optimization.

  • Complex data structures:

    MongoDB can store arbitrary deep nested data structures, but cannot search them efficiently. If your data forms a tree, forest or graph, you effectively need to store each node and its edges in a separate document. (Note that there are data stores specifically designed for this type of data that one should consider as well)

    It has also been pointed out than it is impossible to return a subset of elements in a document. If you need to pick-and-choose a few bits of each document, it will be easier to separate them out.

  • Data Consistency

    MongoDB makes a trade off between efficiency and consistency. The rule is changes to a single document are always atomic, while updates to multiple documents should never be assumed to be atomic. There is also no way to “lock” a record on the server (you can build this into the client’s logic using for example a “lock” field). When you design your schema consider how you will keep your data consistent. Generally, the more that you keep in a document the better.

Mongo schema design (Youtube playlist)

UI Kit

GetUikit

Site: https://getuikit.com/
GitHub: https://github.com/uikit/uikit

A lightweight and modular front-end framework for developing fast and powerful web interfaces.

Well suited for website, landing pages etc.
Simple, straightforward.
Also, nice extra: Autocomplete plugins for both Atom and Sublime Text 3.

Stream Ui Kit

StreamUIkit

Site: https://htmlstream.com/templates/stream-ui-kit

Bootstrap Theme.
So, if you know bootstrap already, you will feel home very fast.

One Nexus

Nexus
Site: http://www.onenexus.io/

More complex
Based on Synergy, a front-end framework for UI components generation. and Kayzen, a custom grid system – All made by the same dev team.

I haven’t take the time to play with this one but it looks nice.

Milligram

Screen Shot 2018 10 17 at 09 40 40
Gotta love simplicity
Only 2kb gzipped
https://milligram.io/

Tabler

One of the best around.
Open source dashboard UI.
Bootstrap based.
Clean code.
Nice logic.

Screen Shot 2018 10 17 at 09 50 01

Site: https://tabler.github.io/
Demo: https://tabler.github.io/tabler/
Source : https://github.com/tabler/tabler

Bootflat

https://bootflat.github.io/

Screen Shot 2018 10 17 at 09 52 20

Semantic UI

Screen Shot 2018 10 17 at 09 55 16

Neat.
Simple.
Working.

https://semantic-ui.com/

PureCSS

Light open source UI Kit

Screen Shot 2018 10 17 at 09 58 16

https://purecss.io/

Ink

Screen Shot 2018 10 17 at 09 59 38

An open source UI kit from Portugal.

http://ink.sapo.pt/

GrounworkCSS

Exist with Rails flavour – or agnostic.

https://groundworkcss.github.io/groundwork/?url=docs/home

Materialize CSS

Screen Shot 2018 10 17 at 10 03 38
Google Material Design
Open source
Not made by Google though.

Site: https://materializecss.com/
Source : https://github.com/Dogfalo/materialize

Also exist as an npm install flavour.

Topcoat

Nice webfrontend for app.
Clean.

Go check it here: http://topcoat.io/

Petal

Made by the folks at Shaker
Site: https://shakrmedia.github.io/petal/
Source: https://github.com/shakrmedia/petal

That’s it.
That’s a good choice to start with.
This doesn’t even include paid one or open source react / vue / angular frameworks.

Let me know if you know of an other one you think should make the list.
These are the one I have found by searching the web for stuff but maybe you know an other one worth mentioning. Hit me by email, in the comments or on social.

Here is a playbook for Ansible to install Docker and Docker Compose on Debian.

You can of course, use a role from Galaxy. You have Gerling’s version or Debops (Ansible stack for Debian).

But here I wanted to implement the install guide from Docker – and translate this into an Ansible playbook (which then should be turn into a custom role for more reusability).

The official guide give use the following install steps:

  1. Update the apt package index
  2. Install packages to allow apt to use a repository over HTTPS
  3. Add Docker’s official GPG key
  4. Add Docker’s official Apt Repository
  5. Then refresh the apt index
  6. Install docker from the package

Yeah, that’s a few steps. But that’s the best way to make sure you always have the last version of docker. Docker is a project that is moving fast and you cannot wait for your favourite distrib to give you the right package, it will always be dirty old version you don’t want. Maybe in some future when things get a bit more stable it will be enough to install it from debian default – but for now that’s really a bad idea.

So, let’s cut the chase here is a the playbook – with comments to make things easy to follow if anything is unclear.


---
- hosts: all
  tasks:
  - name: Update the apt package index
    become: yes
    apt:
      name: "*"
      state: latest
      update_cache: yes
      force_apt_get: yes
  - name: Install packages for apt add repository over HTTPS
    become: yes
    apt:
      name: "{{ packagesdep }}"
      force_apt_get: yes
      state: latest
      update_cache: yes
    vars:
      packagesdep:
      - git
      - apt-transport-https
      - ca-certificates
      - wget
      - software-properties-common
      - gnupg2
      - curl
  - name: Add Apt signing key from official docker repo
    apt_key:
      url: https://download.docker.com/linux/debian/gpg
      state: present
  - name: add docker official repository for Debian Stretch
    apt_repository:
      repo: deb [arch=amd64] https://download.docker.com/linux/debian stretch stable
      state: present
  - name: Index new repo into the cache
    become: yes
    apt:
      name: "*"
      state: latest
      update_cache: yes
      force_apt_get: yes
  - name: actually install docker
    apt:
      name: "docker-ce"
      state: latest

Now this works pretty fine – but we still need to add docker compose – so we do the same – we follow the official documentation – which surprisingly enough do not provide a package but instruct us to curl something – not even with a “latest” tag… so this will require some maintaining – which is annoying but will do for now.

So the step are simple:

  1. Download the latest version of Docker Compose
  2. Add executable permissions to the binary

For this we use the module get_url from Ansible – which replace the curl and allow us to also define the permissions on the fly.


- name: Ensure docker-compose is installed and available
    get_url:
      url: https://github.com/docker/compose/releases/download/1.22.0/docker-compose-{{ ansible_system }}-{{ ansible_userspace_architecture }}
      dest: /usr/local/bin/docker-compose
      mode: 'u+x,g+x'

And that’s it.

As you can see we also made use of {{ansible_system}} and {{ ansible_userspace_architecture }} to replace respectively uname -s and uname -m – if you want a full list of Ansible variables available – you can go check the doc here.

That’s it!

Here is a little research project I did at request to figure out all the gateways, if you want to sell stuff in russian speaking countries – turns out Stripe or Paypal are not exactly going to cover all your needs.

So here is the results of my research on the topic.

Feel free to add in the comments if you have some extra input on the topic.

Yandex Money / Kassa

The first plugin is from Dayes Design :

 

Yandex Money Payment Gateway for WooCommerce

$20 – available on Code Canyon.
Support team behind seems to be available.

Alternative plugin:

 

from Verstayem Vse available here for about 1200 Rb

Now for Yandex Kassa  – and if like me, you had no idea what are the differences between  Yandex Money / Yandex Kassa, and you wanna know you can check this article  (in russian).

Yandex Kassa – which seems to be the official plugin and which seems to work ok if you rely on this demo video:

 

Webmoney

Pretty straight forward, just have a plugin for it :

Wc Webmoney

 

Now if you liked the previous company for yandex money they have one for WebMoney as well:

You can download it here – it costs 1000 rb.

 

PrivatMoney

More geared toward Ukraine, I found this plugin which I haven’t tested – Private24 for less than $20 on webmakers.com.ua – you can buy it here.

Privat woocommerce 600x380

Some alternative free one, which might work but hasn’t been updated in like ages can be downloaded for free here.

QIWI

Ok for Qiwi, I had some trouble finding anything for woocommerce in english, but you can find it from the Verstayem Vse  here.

Now check the other category, because most places I found refered either to Payssion or to Wallet One.

Others

Payssion

You can use intermediate company that provide you right of the bat a middle-man like Payssion (https://www.payssion.com)

and then, they have a plugin you can use that will give you instant access to a large number of payment options: QIWI, Webmoney, Yandex.Money, Sberbank

Wallet one

Walletone seems like a great solution, with a clear pricing scheme (basically a fee+percentage)

Check it here, and they have a plugin as well, which seems pretty cool.

Walletone

Payment supporting:
Visa/Mastercard/Mir
YandexMoney, Webmoney,Qiwi visa Wallet, MasterPass
(check full list here)

If you are in the NGO world and want to collect money for donations you have this interesting and free and maintained plugin Online Leyka

Onlineleyka

From the plugin description :

Supported payment methods includes bank cards Visa and MasterCard through Cloudpayments, PayPal, Chronopay and RBK Money systems, mobile and SMS payments via MIXPLAT, also WebMoney, ROBOKASSA and Yandex.Money are supported. You can also use a traditional bank payment orders (quittances).

 

Now – I haven’t install and double check in production any of these yet, but they all seems pretty legit and seems to work.

If you have other suggestion or precisions I am more than open…

 

On a side note and while we are at it, from personal experience, if you have Stripe set up already (meaning your business / organisation is in an area covered by Stripe) – you will be able – most of the time – to receive payments from customers in russia, but… you will see a higher case of transaction failure – where either the bank or Stripe not letting the transaction happen for seemingly no reason…) but in a pretty good amount of cases – it will work.

 

 

  • Don’t run the internet facing SSH on an IP used by a key application.
  • Use a dedicated host acting as a jump box.
  • Use port knocking.
  • Use key pairs.
  • Use fail2ban.
  • Don’t allow root logins.
  • Restrict access to a named group.
  • Don’t use shared logins.
  • Don’t allow direct access below the presentation tier.

Have more ?

Ping me in the comment or in twitter and I’ll add them here.

Voice and IOT

Amazon Echo, Alexa, Google Home, Apple HomePod and Siri (and Microsoft Cortana) are all fighting to control the voice market. While they get full respect for doing so, I am not gonna say I am particularly trusting any of these companies to provide us with the best level of privacy, service and control over what we do with our devices and how is the data used.

Therefore I am compiling here some resources which I’ll probably update as we go on interesting DIY tools to build similar devices.

Libraries

Cool tutorials

Ok, if you have never heard of Shenzhen, China you might want to get yourself up to date with this quick article – because really, it is getting more and more attention lately, as it has turned into a major platform of electronic hardware manufacturing of the past 15 years.

 

Located north of Hong Kong, Shenzen is the center of the supply chain hub for electronics. All of the major contract manufacturers working for Apple, HP, Microsoft and all the big guys all are there.

 

Shenzhen Electronic Market

This is probably what is the most flashy and amazing part of Shenzhen: the Huaqiangbei electronic market.

A market of spare parts which will have probably anything you could need if you wanted to build any kind of electronic device (check what how this guy has built his own iPhone from spare parts bought on the market)

Visually impressive.

(quick gallery collected by W. Oskay from Evil Mad Scientist Laboratories)

 

And if you want a guided tour – here is one made by Maker’s Muse inside the Shenzhen Electronic Market

 

Hardware Startup Land

Shenzhen is not just about factories and spare parts for geeks – but because of its rich electronics ecosystem, it has become home to a couple of startup accelerators – like HAX :

It makes sense : if you have a hardware startup – better to negotiate directly with the people that will launch your product in mass-production – and since most founders will this hard to do alone, having an infrastructure to guide you is really an attractive proposition for a startup incubator.

Shenzhen: The Silicon Valley of Hardware

This goes beyond spare parts – it is slowly turning into a major place for hardware innovation.

If you want to know more – I suggest you have a look at a piece done by Wired.

They go into making a parallel between the Silicon Valley and how it developed and how Shenzhen is now growing and the differences that comes in.

 

Even if you don’t plan to launch a hardware startup tomorrow, it is one of the places to know about.

 

 

 

 

Ever wonder if you can build your own phone?

Well this guy did.

Last time I checked how to make a phone myself, the only thing I could find always seemed very hard and not very practical (just have a look at this DIY Smartphone based on a rasperry-pi)

But the Strange Parts’s project is really impressive and  give plenty of hopes for a different future of DIY-phones.

While I am probably not going to that myself, it is still rather instructive and inspiring.

Hope you enjoy it!