To Top

Our Blog

Guru Technologies

Vimeo Video Script

The Vimeo Video Script (AKA Maggie Script) is a basic Web Scrapper that retrieves video information from a Vimeo endpoint. This data includes fields such as quality, type, width/height, link, fps, etc. The script will then take the retrieved data and convert it into a CSV file.

While building the application I found that, due to the size of data Vimeo was returning, their Api would occasionally throw a 504 gateway timeout. This error is reasonable as the script was making a request for an array of 100 video objects that contained quite a bit of data. The request would take about 5-10 seconds, and depending on how many videos you want, could be up to 100 requests (The theoretical max). To work around this, I added a 2 second cool down between each request.

Features

  • Multi-threading
  • Settings page (Adjust access token & video count)
  • Progress messages
  • CSV Converter
  • Light Weight (< 1MB)

Images

You can download the executable below

Vimeo Video Download

New Student Classroom Management App – Kodiak Picker

We recently were contacted by a school in Kodiak Alaska to help them do a classroom management app. They had tried unsuccessfully several times to get someone to make this app for them, so we thought we’d take a crack at it. Overall it’s a pretty simple app, but we’re proud of it because we put it together very quickly for them, and they are enthralled with the results. It demonstrates the use of AngularJS and Google’s Material Design that was made originally for the latest Android OS. We made use of the Angular Material plugin on this project.

We released the source code for this project, Kodiak-Picker, under the open source GPL License and the code is freely available on Github: kodiak-picker – Github

The Kodiak Picker app itself is an application written to help track and manages students, quarters/semesters (referred to as “cycles”), and classes (referred to as “tutorials”). The application allows students to choose classes to sign up for out of those available. This can be overriden by teachers or administrators who may know better the needs of specific students. The software can also generate reports about which students are in any class, or search by student to find out what class they are in at any given time. Rooms and instructors are also included for each class. Registration for new cycles can occur while an existing cycle is ongoing.

Kodiak Picker Screenshot

Newest App to Hit the Store…

We just finished our latest iOS app for a longtime client, Inwest Title.

https://itunes.apple.com/us/app/inwest-title/id711322966?mt=8

The app is designed to enhance their current product offerings and give them a competitive advantage in the market. Inwest Title is Utah’s largest independent title company, with 13 office locations. They will now be able to offer features directly on clients phones or tablets such as tracking files, placing orders (even via a picture), quote fees and more.

Inwest75

Startup Weekend Ogden

We decided to take our team to Startup Weekend Ogden in November. According to http://ogden.startupweekend.org/, the event is described as follows:

“All Startup Weekend events follow the same basic model: anyone is welcome to pitch their startup idea and receive feedback from their peers. Teams organically form around the top ideas (as determined by popular vote) and then it’s a 54 hour frenzy of business model creation, coding, designing, and market validation. The weekends culminate with presentations in front of local entrepreneurial leaders with another opportunity for critical feedback.”

After giving a winning pitch, Mike led our team (which included some other awesome entrepreneurs) on a project/application that would be able to facilitate Bitcoin transfers for the banking industry.

Below is a pic of a brainstorming session with our group.IMG_0146

Pictured here are the Guru guys hard at work coding. These guys are incredible;  they did hardcore coding from morning to night the whole weekend! Good thing there was a constant flow of food and drinks provided by the event… IMG_0145-1

At the end of the weekend, we presented our working application! Its amazing what our team accomplished in just 54 hours. We took a concept, designed and built a system, received market validation and have a prototype to show. Go Team!IMG_0151

New Product Release

We have been working with School Improvement Network on enhancing their line of educational products for the last few months. They offer a wide range of tools for professional development and teacher training throughout the United States, Canada, and overseas.

School Improvement Network has just released an app we helped create for them which streamlines the process administrators go through to conduct teacher assessments in the classroom. Check it out here:

Evidence 360

Coming soon from this partnership: a dynamic e-reader teachers can use to continue their professional development.

We created this app using Objective-C.

Troubleshooting Disk Failures on a Linux Software RAID with LVM

The following describes a failure of a drive I had on Ubuntu Linux with a Linux software RAID 5 volume with LVM, how I diagnosed it, and how I went about fixing it. The server had 4 2TB drives in software RAID 5.

When checking kernel messages, here is an example of the bad sectors:

# dmesg -T
[Sun Jul 21 13:36:30 2013] ata4.00: status: { DRDY ERR }
[Sun Jul 21 13:36:30 2013] ata4.00: error: { UNC }
[Sun Jul 21 13:36:30 2013] ata4.00: configured for UDMA/133
[Sun Jul 21 13:36:30 2013] ata4: EH complete
[Sun Jul 21 13:36:32 2013] ata4.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
[Sun Jul 21 13:36:32 2013] ata4.00: irq_stat 0x40000008
[Sun Jul 21 13:36:32 2013] ata4.00: failed command: READ FPDMA QUEUED
[Sun Jul 21 13:36:32 2013] ata4.00: cmd 60/20:00:2c:eb:7e/00:00:08:00:00/40 tag 0 ncq 16384 in
[Sun Jul 21 13:36:32 2013] res 41/40:00:2f:eb:7e/00:00:08:00:00/40 Emask 0x409 (media error) &lt;F&gt;
[Sun Jul 21 13:36:32 2013] ata4.00: status: { DRDY ERR }
[Sun Jul 21 13:36:32 2013] ata4.00: error: { UNC }
[Sun Jul 21 13:36:32 2013] ata4.00: configured for UDMA/133
[Sun Jul 21 13:36:32 2013] sd 3:0:0:0: [sdd] Unhandled sense code
[Sun Jul 21 13:36:32 2013] sd 3:0:0:0: [sdd] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Sun Jul 21 13:36:32 2013] sd 3:0:0:0: [sdd] Sense Key : Medium Error [current] [descriptor]
[Sun Jul 21 13:36:32 2013] Descriptor sense data with sense descriptors (in hex):
[Sun Jul 21 13:36:32 2013] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[Sun Jul 21 13:36:32 2013] 08 7e eb 2f
[Sun Jul 21 13:36:32 2013] sd 3:0:0:0: [sdd] Add. Sense: Unrecovered read error - auto reallocate failed
[Sun Jul 21 13:36:32 2013] sd 3:0:0:0: [sdd] CDB: Read(10): 28 00 08 7e eb 2c 00 00 20 00
[Sun Jul 21 13:36:32 2013] end_request: I/O error, dev sdd, sector 142535471
[Sun Jul 21 13:36:32 2013] ata4: EH complete

It’s clear in this case with those messages that the problem was with sdd, but in some cases, it’s not always clear which ata# in dmesg matches up with which drive, so for that I followed this askubuntu guide to see if ata4.00 was the same as /dev/sdd:

In my case, it’s clear that the problem is indeed with /dev/sdd or (ata4)

I have my RAID configured to email me if there’s a problem with the RAID, but it wasn’t doing it, even though I confirmed it was configured to do so.

I checked to make sure the RAID looked normal:

cat /proc/mdstat

looked normal (all U’s on the drives)

The output of mdadm looked fine as well

# mdadm --detail /dev/sdd

The problem from dmesg seems to be that it was having trouble reading specific sectors on drive [sdd]. So I knew I needed to start checking that drive specifically.

There are some good guides available about dealing with badblocks, such as the “bad block HOWTO”

Also, the FAQ for smartmontools (which contains the smartctl program):

The problem with the bad block HOWTO was that didn’t cover my case which is RAID/LVM. You definitely don’t want to get the dd command wrong, and I was nervous about trying to get it right on my system. A better approach my be to use hdparm as described here (forcing a hard disk to reallocate bad sectors). However, I didn’t do this for reasons explained below

The crux of that last page is looking for the bad sector in dmesg, and then confirming is with this command (entering your sector):

# hdparm –read-sector 1261069669234239432572396425

You should get:

/dev/sdb: Input/Output error

Then, the drive can’t be part of the array when you do this, doing a:

# hdparm –write-sector 1261069669234239432572396425 /dev/sdb

Followed by an force assemble to get the drive back in.

However, I didn’t want to do all that for two reasons, I didn’t know the extent of the bad sectors (if the bad sectors were isolated, or the whole drive was going bad), and because the command writes 0’s into the sector which means you will loose data if not done carefully (taking precautions to ensure that since our array is still functioning otherwise, I don’t write 0’s into the good drives), so I decided to try something else instead.

I started with smartctl reported that the individual drive was healthy. It turns out that a drive can actually be on the verge of failure or not healthy, and smart still reports it as being healthy. Smart health is more like an indicator that things can go bad not a definitive way to tell:

# smartctl -Hc /dev/sdd
SMART overall-health self-assessment test result: PASSED

Then I checked for vendor-specific SMART attributes using -a. The thing that stands out here is the “197 Current_Pending_Sector” being higher than 0 (in my case 40), meaning that there are sectors pending re-allocation. This means there is a high likelihood there are bad sectors.

# smartctl -a
 
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 253 253 021 Pre-fail Always - 6200
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 23
5 Reallocated_Sector_Ct 0x0033 188 188 140 Pre-fail Always - 89
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 093 093 000 Old_age Always - 5693
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 21
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 18
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 4
194 Temperature_Celsius 0x0022 080 074 000 Old_age Always - 72
196 Reallocated_Event_Count 0x0032 113 113 000 Old_age Always - 87
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 40
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0

I decided to run a short test:

# smartctl -t short /dev/sdd

And then 60 seconds later I checked the status:

# smartctl -l selftest /dev/sdd
 
=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 2 Short offline Completed: read failure 90% 5691 71314612

That confirmed there was a bad sector. At this point, I decided I needed to make a judgment on whether the drive was going bad, or there were just some bad sectors I needed to mark as such and move on. You can mark sectors as bad by writing 0’s to the bad area, and the disk firmware should automatically mark them as bad. The reason for this is explained in the earlier mentioned smartmontools FAQ:

If the disk can read the sector of data a single time, and the damage is permanent, not transient, then the disk firmware will mark the sector as ‘bad’ and allocate a spare sector to replace it. But if the disk can’t read the sector even once, then it won’t reallocate the sector, in hopes of being able, at some time in the future, to read the data from it. A write to an unreadable (corrupted) sector will fix the problem. If the damage is transient, then new consistent data will be written to the sector. If the damange is permanent, then the write will force sector reallocation. Please see Bad block HOWTO for instructions about how to force this sector to reallocate (Linux only).

The disk still has passing health status because the firmware has not found other signs of trouble, such as a failing servo.

Such disks can often be repaired by using the disk manufacturer’s ‘disk evaluation and repair’ utility. Beware: this may force reallocation of the lost sector and thus corrupt or destroy any file system on the disk. See Bad block HOWTO for generic Linux instructions.

The problem I had is the “bad block HOWTO” guide I mentioned earlier doesn’t cover my scenario, RAID/LVM. I’m sure you could dig in and find exactly the sector and mark it, but I didn’t want to risk it. So I was about to track down a western digital disk evaluation and repair utility, when I ran across a post that suggested I can just do a RAID sync (was a “repair” on older kernels”). To initiate, you run:

# echo 'check' > /sys/block/md0/md/sync_action

Then check the RAID check status with:

# cat /proc/mdstat

In my case, it was going really slow, so I first did what I could to shut down unecessary activity on the drive, and then ran through suggestions from here

The main thing that sped things up was setting the stripe cache size to a higher level than the default 256.

# echo 32768 > /sys/block/md3/md/stripe_cache_size

As it was doing the check, lots of errors were being thrown about the drive in dmesg, so I knew it wasn’t an isolated incident I was going to be able to fix by using a drive utility to mark bad sectors, the whole drive would need to be replaced.

As I was monitoring the RAID status, it got through about 5% and then the RAID removed the drive from the array and stopped it’s work. Here’s what dmesg said:

[Sun Jul 21 17:14:29 2013] md/raid:md0: Disk failure on sdd2, disabling device.
[Sun Jul 21 17:14:29 2013] md/raid:md0: Operation continuing on 3 devices.
[Sun Jul 21 17:14:29 2013] md: md0: data-check done.

so now I have a degraded array, and need to get a new drive ASAP to replace it and rebuild the array.

To do this, I had to ensure that the serial numbers matched properly.  Also, since the drive was no longer showing up in my system, I had to issue one of the following commands:

mdadm /dev/md0 -r detached

or

mdadm /dev/md0 -r failed

The man page says:

“The first causes all failed device to be removed. The second causes any device which is no longer connected to the system (i.e an ‘open’ returns ENXIO) to be removed. This will only succeed for devices that are spares or have already been marked as failed.”

Keep command-line Subversion client (svn) from saving credentials

If you share a server with multiple users, you may find yourself committing as root.  In that case, it’s helpful if you can keep svn from saving credentials.  All you have to do is edit:

/root/.subversion/config

Set the following:

store-passwords = no
store-auth-creds = no

Then remove any credentials file that exists in:

/root/subversion/auth

Separating a foreground object from its background using GIMP’s Color to Alpha feature

Recently I was trying to separate a foreground image from its background so I could put the image on a new background of a completely different color.  There is a GIMP tutorial, but it failed to help me remove the ugly white or colored edges from the foreground.  Here is the image I started with:

Using that above mentioned GIMP tutorial, I tried my best to isolate only the text and place it on my background.  The best I could do was this:

The Solution

After using the Fuzzy Select tool to isolate the foreground image from it’s background (in this case, a white background) and pasting it into a new layer, I then set the entire background layer use the dark red color I wanted.  Then I used the “Color to Alpha” feature from the “Colors” menu.  It brings up this dialog:

From here, you specify the color you want made into alpha.  I left it at the default, white.  According to the GIMP documentation, this tool, “…will attempt to preserve anti-aliasing information by using a partially intelligent routine that replaces weak color information with weak alpha information. In this way, areas that contain an element of the selected color will maintain a blended appearance with their surrounding pixels”.  That is exactly what I needed.

Color to Alpha Applied

I had one final thing I wanted to do to the image, which was change the black color to a different one.  For this, I used the “Colorize” tool on the top layer, also in the “Colors” menu.

The Result

     And the zoomed out, final version:    

Return JSONP with CakePHP

Here’s a nice way to turn on JSONP for CakePHP. JSONP, or JSON with padding, is necessary when you want to include JSON from a site different than the one you’re on.

A good explanation for what JSONP is can be found here: http://stackoverflow.com/questions/2067472/what-is-jsonp-all-about

Just include this code on any CakePHP controller that you will be using with JSONP (or you could put it in the AppController to make it work everywhere in your app but that may add a slight performance hit). It checks to make sure that JSON is being returned before trying to apply the padding. This code overrides the afterFilter() which is the last controller action CakePHP does after doing rendering.

    // This funciton makes so JSONP works
    public function afterFilter() {
      parent::afterFilter();
 
      if (empty($this->request->query['callback']) || $this->response->type() != 'application/json') {
          return;
      }
 
      // jsonp response
      App::uses('Sanitize', 'Utility');
      $callbackFuncName = Sanitize::clean($this->request->query['callback']);
      $out = $this->response->body();
      $out = sprintf("%s(%s)", $callbackFuncName, $out);
      $this->response->body($out);
    }

Web browser text search

So I am working on a project where I have to search some HTML documents for a possible text string. This lead me to an interesting find in how browsers do searching on text.

 

Let’s look at the following HTML.

<div>
<div style="background-color: #00ff00;">Hello ther</div>
<div style="background-color: #ff0000;">e,
<span style="background-color: #f0ff00;">bob</span> how are</div>
<div style="background-color: #00ff00;">you?</div>
</div>

The text displays as the following: Text displayed in a browser

The yellow represents the inner span. The divs are different colors to show their distinction.

If you search for the text “Hello there” no results will be found as the browser does not search across div elements.

If you search for the text “e, bob how are” the text will be found as the browser will search across span elements.

It’s interesting to note that the browser’s search works only across defaulted inline elements but not elements set inline by the html.