Yaoyu Yang Homepage

How to add a list of Amex offers without clicking one at a time?

Sat, 02 Sep 2023 18:00:00 +0000

The Problem

American Express credit cards come with Amex offers. You can read this guide or this guide in Chinese if you want to know more about Amex offers. The way to add the Amex offer is to log into the Amex website get to the UI like the following and click the button to add the offer one at a time. It’s getting crazy when you have say 100 offers and you have to click 100 times to add all the offers. Can we do it with one click instead?

A Solution

A solution I came up with is to run the following Javascript oneliner script in your browser console. For Chrome, you can get to the console by going to More Tools > Developer Tools. You can copy the following script and paste into the console and press return on your keyboard. Make sure you log into the Amex website and click View All to load all the offers before running the script.

offers = Array.from(document.querySelectorAll("button[title='Add to Card']"))
for (const offer of offers) {
  offer.click();
  await new Promise(r => setTimeout(r, 2000));
}

After you run the script, you should see all added offers appear in the Added to Card tab. You might need to do this again if you have more than 100 offers as Amex only loads 100 offers at a time.

An even better solution using Bookmarklet

So I showed this script to my friend Henan Li and he came up with an even better solution utilizing this script with Bookmarklet. This allows you to run the script with one click on the bookmark section. This is easier than going to the developer console and running code. Follow the steps below:

Create a new bookmark in Chrome by clicking on the bookmark section -> Add Page. If you use other browsers, just add a new bookmark following other browsers’ UI.
Enter a name you like (e.g. I name it Add Amex offer), and enter the following line in the URL section.

javascript:offers=Array.from(document.querySelectorAll("button[title='Add to Card']"));c=()=>{ offer = offers.pop(); if (!offer) return console.log('Added all offers!'); offer.click(); setTimeout(c, 2000) };c();

Go to the Amex website, log in to your account, load all the offers by clicking View All, click this bookmark, wait a bit and all your offers will be added automatically. You might need to do this again if you have more than 100 offers as Amex only loads 100 offers at a time.

How it works

The script uses document.querySelectorAll to find all the elements that match CSS selector pattern of the Add to Card button, turn them into an Array, and then apply .click() on each element to click the button. By running the script, you are basically using the script to click the buttons for you and this saves you from manually clicking the buttons one at a time.

Shortcoming

This script depends on how the button is built in the Amex UI. If Amex updates the UI and the CSS selector pattern of button[title='Add to Card'], then this script will not work. However, I’ll try to update this script here if Amex updates its UI and changes the CSS selector pattern.

Disclaimer

THE SCRIPT ABOVE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SCRIPT OR THE USE OR OTHER DEALINGS IN THE SCRIPT.

Jira/Gitlab Automation at Ginkgo

Fri, 17 Jun 2022 18:00:00 +0000

Check out the contents of this blog post at Ginkgo Bioworks, where I discussed benefits of using Jira/Gitlab automation at Ginkgo.

Solution Engineering at Ginkgo Bioworks

Tue, 01 Dec 2020 18:00:00 +0000

Check out the contents of this blog post at Ginkgo Bioworks, where I discussed Solution Engineering team at Ginkgo and my experience as a Solution Engineer.

Solving a database concurrency problem in a Django based genome tracking software

Sat, 09 Nov 2019 18:00:00 +0000

The Problem

Recently I’ve been working on a bug fix for an open source software called Edge, which is used in Ginkgo Bioworks for tracking genome changes efficiently. Edge keeps structural changes between a genome and child genomes derived from it. Edge does not store child genomes as brand new genomes and only store the changes, therefore is very space efficient. The bug is related to manage concurrency when making genome changes. Previous version of the code does lock the genome that is subject to change. The code for database lock is here.

locked_genome = Genome.objects.select_for_update().filter(pk=genome.id)
locked_genome = list(locked_genome)
genome = locked_genome[0]

If you look at this code the first time, you would think this code already solved the concurrency problem since we locked the genome we are about to make change. However, if you look into Edge’s data structure, you will realize this lock may work for some cases but opens doors for potential data corruption. The reason is the following: Edge stores Genome sequences as a linked list of Chunks. During genome change event, for example, integrating a new 50 bp sequence into the genome, the chunks are usually broken down into smaller Chunks to make tracking and storing sequences easier. The trick is that child, parent, and parent’s ancestor genomes will all share these same smaller chunks for the parts that are the same across them. Therefore, during a genome change event, we are not only modifying the parent and child genome data, we are also modifying all parent’s ancestors genomes because we are updating how their shared sequence are stored and represented in Chunks. Therefore, if two genomes share the same Chunks are concurrently performing genome changes, since we only lock the genome row, the Chunks rows could overwrite each other and resulting in unexpected changes.

The Solution

The solution we came up with is actually pretty neat. Instead of locking the genome we are about to change, we lock the root genome. Any other genomes that share the same Chunks are going to have the same root genome, therefore, they will need to lock the same root genome if they were to perform the change at the same time. Thus, we can issue the concurrent genome changes safely. This will allow us to perform true parallel genome changes when genomes don’t share the same root genome while avoiding possible data corruption if we are performing parallel genome changes for genomes that do share the same root genome. We coded up this solution in this Pull Request, where we added a function to find the root genome:

def find_root_genome(genome):
    root_genome = genome
    while root_genome.parent_id:
        root_genome = Genome.objects.get(pk=root_genome.parent_id)
    return root_genome

and a line of code to lock the root genome whenever we are starting to make the genome change in database.

Genome.objects.select_for_update().filter(pk=find_root_genome(genome).id)

The Gotchas

We thought the above code would solve the problem, however, after we deployed this change to staging, we did not find the root genome locked during the genome change transaction. The trick we used to find that the root genome was not locked was to add a print statement after the lock code, a 10 seconds delay, and another print statement just before the transaction finishes.

import time
Genome.objects.select_for_update().filter(pk=find_root_genome(genome).id)
print(f'Start genome change for {genome.id}')
time.sleep(10)
...
print('Genome change for {genome.id} about to finish')

We issue genome changes to g1 and g2 that share the same root genome g0 at about the same time, we would expect the sets of print statements from two genome changes one after another, not interleaving. Basically, we would expect the log looks like:

Start genome change for g1
Genome change for g1 about to finish
Start genome change for g2
Genome change for g2 about to finish

However, we actually saw:

Start genome change for g1
Start genome change for g2
Genome change for g1 about to finish
Genome change for g2 about to finish

This means that the lock on root genome g0 did not happen. So we started to take another look at the Django doc for select_for_update and this text When the queryset is evaluated (for entry in entries in this case), all matched entries will be locked made us realize that just doing

Genome.objects.select_for_update().filter(pk=find_root_genome(genome).id)

will actually not lock the root genome since the queryset is not actually evaluated. Then we went on and updated the code in this Pull Request. Lock only happens when queryset is evaluated, therefore need to do at least genomes[0].

genomes = Genome.objects.select_for_update().filter(pk=find_root_genome(genome).id)
genome = genomes[0]

After this change, the log looks as expected when we make two concurrent genome changes that share the same root genome. Problem solved!

Things we learned:

One trick to solve the concurrency problem is to lock common resources during database traction, even the common resources are not the resources to be updated in the transaction. Here we used this trick to find the root genome that is common and locked it.
select_for_update lock only happens when the Django queryset is evaluated.

P.S. My team at Ginkgo Bioworks is hiring for dev positions at all levels. If you are interested in building software to engineer biology to make a dent in the universe, check out here!

Deploying an AngularJS app to Amazon EC2 (Part II)

Thu, 15 Jun 2017 12:00:00 +0000

Introduction

A few months ago, I deployed an AngularJS app called SeafoodCheck on an Amazon EC2 instance. It wasn’t a trivial task and I had to go through various tutorial and stackoverflow pages to get it working. To help people in similar positions as I was, I created this “Deploying an AngularJS app to Amazon EC2” tutorial.

This serves as the Part II of this tutorial aiming to provide a step by step guide in running an Angular JS app on an Amazon EC2 instance. If you don’t know how to set up an Amazon EC instance, Part I of the tutorial published ealier described how to set up an Amazon EC2 instance.

Prerequisites

An AWS EC2 instance set up
Basic Git knowledge and a working Github account. Hello World on Git and Github if you need.
Basic command line knowledge
Basic knowledge of Node.js

Fork AngularJS app code

For this tutorial, we’ll be using the PhoneCat Tutorial App from AngularJS. This is the offical tutorial from AngularJS. I highly recommend you to go through if you are starting on AgularJS.

To begin, let’s fork PhoneCat Tutorial App on Github. You should then have a repo under your name like this forked from angular/angular-phonecat.

SSH into EC2 and git clone code

Now SSH into your EC2 instance. Type ls to see the directories. You should see something like the following: Now do the following:

cd apps
sudo chown -R bitnami /opt/bitnami
sudo chown -R bitnami /home/bitnami
git clone https://github.com/yaoyuyang/angular-phonecat {USE YOUR LINK}

The chown steps are very important since the default owner for these two directories, which will be used later to write files, are root. If you don’t change the ownership, you have to install everything under root and a lot of permission problems will pop up later.

Run the app

Before you run the app, edit the package.json using your favorite editor. Replace localhost with 0.0.0.0. This step and the Security Access steps in the next section expose your app to the public internet.

Now you can run the app by the following:

cd angular-phonecat
npm start

You should see something in the terminal like:

Starting up http-server, serving ./app
Available on:
  http://127.0.0.1:8000
  http://172.31.6.114:8000 (This will be your private IPs)

Security Access

Your Angular app is now running. However, you need to configure the security access on AWS before you can let the world to see your app for real.

Open up your AWS EC2 dashboard, click Security Groups on the left panel. Click on the row that has Group name MEAN powered by Bitnami. Then click the inbound tab beneath. We need to customize the rules to make port 8000 accessible. Click Add Rule, leave Protocol default as TCP, in Port Range enter 8000, and Source choose Anywhere. Click Save. Refresh and you should see the following:

Check out your running app on browser

After we give the security access. We can now access our app through a brower. Type in your Public DNS:8000. In my case it’s http://ec2-34-210-41-187.us-west-2.compute.amazonaws.com:8000. Hooray! You should now see the Google Phone Gallery up!

Forever run

To keep the app running continuouly without being affected by logging out of EC2 instance. We use the node package forever.js. First ctrl+c to stop the existing app running. Install the package:

sudo npm install forever -g

And then start the app by:

forever start ./node_modules/http-server/bin/http-server ./app -p 8000 -d false

To stop the app. Use forever list then forever stop with the id, e.g. forever stop 12305.

Deploying an AngularJS app to Amazon EC2 (Part I)

Sat, 15 Apr 2017 12:00:00 +0000

Introduction

This serves as the Part I of this tutorial aiming to provide a step by step guide in getting an Amazon EC2 instance set up for those who have little experience of AWS. It can also be served as a stand alone tutorial for those who are just interested in learning how to create an AWS account and getting an EC2 instance setup.

Prerequisites

Know how to execute commands in a shell

Setting up Amazon EC2

Amazon EC2 (Elastic Compute Cloud) is a virtual computer in the cloud that can run user’s own computer applications. EC2 is one of the most popular services provided by Amazon Web Services (AWS). AWS provides a large list of services related to cloud computing. With the right environment setup, we can run our AngularJS app on an Amazon EC2 instance. Let’s set up our Amazon EC2. There are two steps to set up an EC2 instance if you are new to AWS.

Create an AWS account

First, you need to create an AWS account. Follow the following steps.

Go to AWS and click Create an AWS account.
Type in your email, choose I am a new user, and click Sign in using our secure server.
Fill out your name and password and click create account
Choose Personal Account and fill out required information
Fill out credit card info. You won’t be charged for the first 12 months if you fall into the free tier. Complete this tutorial should not get your charged at least for the first 12 months if you are new to AWS and only run this instance.
Complete Indentity Verification by providing a valid phone number for an automated phone call where you will need to enter in a 4 digit code shown on the screen.
In the Support Plan, choose Basic so there won’t be extra charge.
Click Complete Sign Up. It will redirect you to the login page and now you should see the following AWS home page.

Create an EC2 instance and connect to it

Second, let’s create an EC2 instance.

Click Services on the top left menu, then click EC2
Create a Key Pair by first clicking Key Pairs in the NETWORK & SECURITY section on the left verticl menu. Then Create Key Pair, type in your app name and click Create. Save the .pem file. Please don’t lose this file. You have only this chance to download it and you’ll need it to access your EC2 instance. Also note which region you created your key in. It’s shown on the top right of the navigation bar near Support. The default region for me was Oregon.
Go to AWS Marketplace, in the search bar, type Bitnami MEAN image, click on the search results, and you should see the following. Choose a region that you created your key in (mine was Oregon) and click Continue.
Choose 1-Click Launch, make sure you are selecting t1.micro for the EC2 Instance Type. Click Accept Software Terms & Launch with 1-Click.
Now go back to your AWS management console, click EC2, and you should see that you have 1 running instance. You have successfully created an EC2 instance! Click Connect button near Lanuch Instance button. AWS provides you a step by step instruction for how to connect to this instance using ssh, providing that you have access to an Unix or Unix-like shell. Follow it and you should be able ssh into your instance on AWS! Check this tutorial if you have a PC running Windows.
Now you should be connected to your EC2 instance with a fully loaded Linux machine at your disposal! Congratualations on acheiving this step!

How to read multiple files in Node.js asynchronously and process all file contents synchronously?

Fri, 20 Jan 2017 12:00:00 +0000

The problem arose when I was working on a web application using Node.js and Angular JS, where I need to read multiple files in a directory, process the contents in all files aggregately, and write processed content into a new file. This problem is similar to this stackoverflow question (“Reading all files in a directory, store them in objects, and send the object in node.js”).

The specific problem I encountered was as follows. I have a list of json files in a directory each describe details of a fish with id, images, name, etc.

/fish1.json

{
    "id": "Cod-Alaskan",
    "name": "Cod (Alaskan)",
    "images": [
        "img/fish/Cod (Alaskan)1.jpg",
        "img/fish/Cod (Alaskan)2.jpg"
    ],
    "mercury": 0
}

/fish2.json

{
    "id": "Sole-Pacific",
    "name": "Sole (Pacific)",
    "images": [
        "img/fish/Sole (Pacific)1.jpg",
        "img/fish/Sole (Pacific)2.jpg",
    ],
    "mercury": 1
}

…

I want to process all these files and write to a summary file like this: /fishes.json

[
    {
        "name": "Cod (Alaskan)",
        "mercury": 0,
        "imageUrl": "img/fish/Cod (Alaskan)1.jpg",
        "id": "Cod-Alaskan"
    },
    {
        "name": "Sole (Pacific)",
        "mercury": 1,
        "imageUrl": "img/fish/Sole (Pacific)2.jpg",
        "id": "Sole-Pacific"
    },
    ...
]

Due to the async nature of Node.js, simply using fs.readFile would result in all files processed asynchronously. How can we know when all the reading has completed so we can start processing the results? For processing a single file, one could use callback function in javascript to achieve this easily, but for multiple files, it is not a good style as pointed out by this stackoverflow answer. As suggested by this answer, one could either use sync versions of fs functions or Promises. Using sync versions of fs functions would block each file reading and possibly result in slow reading if you have a large number of files. Here I present the code snippet for using Promises to solve this problem.

var fs = require('fs');

// make Promise version of fs.readdir()
fs.readdirAsync = function(dirname) {
    return new Promise(function(resolve, reject) {
        fs.readdir(dirname, function(err, filenames){
            if (err) 
                reject(err); 
            else 
                resolve(filenames);
        });
    });
};

// make Promise version of fs.readFile()
fs.readFileAsync = function(filename, enc) {
    return new Promise(function(resolve, reject) {
        fs.readFile(filename, enc, function(err, data){
            if (err) 
                reject(err); 
            else
                resolve(data);
        });
    });
};

// utility function, return Promise
function getFile(filename) {
    return fs.readFileAsync(filename, 'utf8');
}

// example of using promised version of getFile
// getFile('./fish1.json', 'utf8').then(function (data){
//     console.log(data);
// });


// a function specific to my project to filter out the files I need to read and process, you can pretty much ignore or write your own filter function.
function isDataFile(filename) {
  return (filename.split('.')[1] == 'json' 
          && filename.split('.')[0] != 'fishes'
          && filename.split('.')[0] != 'fishes_backup')
}

// start a blank fishes.json file
fs.writeFile('./fishes.json', '', function(){console.log('done')});


// read all json files in the directory, filter out those needed to process, and using Promise.all to time when all async readFiles has completed. 
fs.readdirAsync('./').then(function (filenames){
    filenames = filenames.filter(isDataFile);
    console.log(filenames);
    return Promise.all(filenames.map(getFile));
}).then(function (files){
    var summaryFiles = [];
    files.forEach(function(file) {
      var json_file = JSON.parse(file);
      summaryFiles.push({ "name": json_file["name"],
                          "imageUrl": json_file["images"][0],
                          "id": json_file["id"]
                      });
    });
    fs.appendFile("./fishes.json", JSON.stringify(summaryFiles, null, 4), function(err) {
        if(err) {
          return console.log(err);
        }
        console.log("The file was appended!");
    });
});

How to DIY internationalize/translate your iOS app?

Thu, 22 Sep 2016 12:00:00 +0000

I have been working on an iOS app called SeafoodCheck primarily designed for pregnant women to check mercury level in fish with data from FDA. Recently, I have feedback from users in China asking for Chinese translations of the app. It seems a straightforward process but it turns out there is not a super good tutorial to guide you through the process. During my learning process, I figured out a pretty neat way to add these translations based on this simple framework called Localize-Swift.

Simple steps to add Travis CI to your Pebble project repo on Github

Sat, 12 Mar 2016 12:00:00 +0000

I have been working on a Pebble watch app called CatchOneBus for almost a year. The project’s source code is hosted on Github and I want to add integration with Travis CI to have that cool icon light up! Even though lighting up the icon is my biggest motivation for now, Travis CI is way more than an icon for sure. Travis CI is a continuous integration service used to build and test software projects hosted at GitHub. Simply speaking, it can run your project on a machine in the cloud as if you are running the project on your local machine and tell you if it can run successfully or not. Using Travis also gives you the confidence that it has been tested on other machines other than your local machine.

Anyway, here are the steps for add Travis CI in your Pebble app project repo:

Add Travis CI integration to your Github by clicking Add to Github.
Head to your Travis CI profile page and flick the repository switch on for your pebble project repo.
Add a .travis.yml into your Github Pebble project repo. You open this and copy or directly copy the following code snippet into your .travis.yml file.

sudo: false
language: python
python:
  - "2.7"

env:
  - PEBBLE_SDK_VERSION=3.10.1

before_install:
  - wget https://s3.amazonaws.com/assets.getpebble.com/pebble-tool/pebble-sdk-4.2.1-linux64.tar.bz2
  - mkdir -p ~/.pebble-sdk
  - tar -jxf pebble-sdk-* -C ~/.pebble-sdk
  - touch ~/.pebble-sdk/ENABLE_ANALYTICS
  - export PEBBLE_SDK=~/.pebble-sdk/pebble-sdk-*
  - export PEBBLE=$PEBBLE_SDK/bin/pebble

install:
  - pushd $PEBBLE_SDK
  - virtualenv --no-site-packages .env
  - source .env/bin/activate
  - pip install -r requirements.txt
  - deactivate
  - popd
  - $PEBBLE sdk set-channel beta
  - yes | $PEBBLE sdk install $PEBBLE_SDK_VERSION

script:
  - $PEBBLE build

Trigger your first build with a git push.
Head over to Travis CI. Once it’s finishing building , the text and icon color will turn to green. Click the icon next to the icon. Select MARKDOWN and copy the code into your README.md file.
You should now see the cool icon light up in your project repo!

Starting to blog and code snippets to determine leap year!

Mon, 29 Feb 2016 00:00:00 +0000

It's the last day of February in 2016. This year is a leap year so there is one extra day in February. The algorithm to determine a leap year is as follows: A year will be a leap year if it is divisible by 4 but not by 100. If a year is divisible by 4 and by 100, it is not a leap year unless it is also divisible by 400. It's a very simple algorithm, a very "naive" Ruby function to determine leap year will be like:

def leap_year? year
  if year%4 == 0
    if year%100 != 0
      true
    elsif year%400 == 0
      true
    else
      false
    end
  else
    false
  end
end

Above code will give correct answer but it looks quite verbose, though a very good exercise using if, else and elsif. A more succinct code snippet looks like following:

def leap_year? year
  (year%4 == 0 && year%100 != 0) || (year%400 == 0)
end

The logical operators are magic! Also noting that the above code uses the implicit return in Ruby, whatever got evaluated last in the function got returned. This is a nice feature of Ruby in someway, saving you a few keystrokes and makes the code more like a human written paragraph of an article other than a piece of computer code. Whereas in Python, you have to explicitly write return, so the code for calculating leap year in python will look like:

def leap_year(year):
  return (year%4 == 0 and year%100 != 0) or (year%400 == 0)

I use Ruby and Python a lot during everyday work and research, love both of them and always willing to learn more. The aim of this blog is to share my learning process and some knowledge of various kinds of technologies including but not limited to programming, app development, synthetic biology, etc. Looking forward to write more and learn more!