This feed contains pages with tag "python".

Python Background

Octave Background

Most of the octave material here is covered in the Gnu Octave Beginner's Guide, in particular

GOBG Chapter 2
GOBG Chapter 4
GOBG Chapter 5

For some parts, we will refer to the GNU Octave Manual

We'll refer to the online text by Robert Beezer for linear algebra background. We can happily ignore the stuff about complex numbers.

Before the lab

One of the main features of Octave we will discuss is vectorization. To understand it, we need some background material on Linear Algebra. If you've taken a linear algebra course recently, this should be easy, otherwise you will probably need to review

Vector Operations, particularly
- Addition
- Subtraction
- Scalar multiplication
Inner product
Matrix Operations
- Addition
- Subtraction
- Scalar multiplication
Matrix Vector Product
Matrix Multiplication

If you don't have time before this lab, make sure you review any needed linear algebra before the next lab.

Counter, generators, and classes

Time: 15 minutes
Activity: Demo

Recall our generator based counter from L17.

def make_counter(x):
    print('entering make_counter')
    while True:
        yield x
        print('incrementing x')
        x = x + 1

In Lab 17 we almost simulated the generator behaviour with a closure, except that next(counter) was replaced by counter(). We can provide a compatible interface by using a python class.

class Counter:
    "Simulation of generator using only __next__ and __init__"
    def __init__(self,x):
        self.x = x
        self.first = x

    def __next__(self):




        self.x = self.x + 1
        return self.x - 1

print('first')
counter = Counter(100)
print('second')
print(next(counter))
print('third')
print(next(counter))
print('last')

Observe that the implimentation is closer to the closure based version of L17 than the original generator version.
It's a recurring theme in Python to have some syntax/builtin-function tied to defining a special __foo__ method

Fibonacci, again

Time: 35 minutes
Activity: individual

Save the generator based Fibonacci example as ~/fcshome/cs2613/L18/fibgen.py

Save the following in ~/fcshome/cs2613/L18/fib.py

#!/usr/bin/python3
class Fib:
    def __init__(self,max):
        self.max = max
        self.a = 0
        self.b = 1

    def __next__(self):
        if self.a < self.max:



        else:
            raise StopIteration

Complete the definition of __next__ so that following test passes. Hint: Consider following the closure based version from Lab 17

from fib import Fib
from fibgen import fibgen

def test_fib_list():

    genfibs=list(fibgen(100))
    fibber=Fib(100)

    fibs=[]
    while True:
        try:
            fibs.append(next(fibber))
        except:
            break

    assert genfibs == fibs

We may wonder why this odd looking while loop is used to build fibs rather than some kind of list comprehension. The following simpler version currently fails:

def test_fib_list_2():
    genfibs=list(fibgen(100))
    classfibs=list(Fib(100))
    assert genfibs==classfibs

This failure is due the fact that we haven't really built an iterator yet. Remember Python works by duck-typing, which means that an iterator is something that provides the right methods.

Add the following method to your Fib class. Observe the test above now passes.

    def __iter__(self):
        return self

In addition to signal to the Python runtime that some object is an iterator, the __iter__ serves to restart the traversal of our "virtual list". If we want iterators to act as lists, then the following should really print the same list of Fibonacci numbers twice

if __name__ == '__main__':
    fibber = Fib(100)
    for n in fibber:
        print(n)
    for n in fibber:
        print(n)

Since it doesn't, let's formalize that as a test. Modify the __init__ and (especially) the __iter__ method so that following test passes.

def test_fib_restart():
    fibobj = Fib(100)
    list1 = list(fibobj)
    list2 = list(fibobj)
    assert list1 == list2

Running Octave

There is a GUI accessible from Activities -> GNU Octave, or by running from the command line
```
  % octave --gui
```
There is also a REPL accessible from the command line by running
```
  %  octave
```
To sanity check your octave setup, run the following plots
```
  >> surf(peaks)
  >> countourf(peaks)
```

Fibonacci

Time: 15 minutes
Activity: Demo/Group programming.

Let's dive in to programming in Octave with a straight-forward port of our Python Fibonacci function. Save the following in ~/fcshome/cs2613/labs/L19/fib.m. It turns out to be important that the function name is the same as the file name.

function ret = fib(n)
  a = 0;
  b = 1;
  for i=0:n



  endfor
endfunction

%!assert (fib(0) == 0);
%!assert (fib(1) == 1);
%!assert (fib(2) == 1);
%!assert (fib(3) == 2);
%!assert (fib(4) == 3);
%!assert (fib(5) == 5);

We can avoid the need for a temporary variable by calling deal function, but it's not clear that would be faster. If you have time, try it.
Note the %! assert. These are unit tests that can be run with
```
  >> test fib
```
The syntax for %!assert is a bit fussy, in particular the parentheses are needed around the logical test.

Fibonaccci as matrix multiplication

Time: 20 minutes
Activity: individual

The following is a well known identity about the Fibonacci numbers F(i).

[ 1, 1;
  1, 0 ]^n = [ F(n+1), F(n);
               F(n),   F(n-1) ]

Since matrix exponentiation is built-in to octave, this is particularly easy to implement in octave

Save the following as ~/fcshome/cs2613/labs/L19/fibmat.m, fill in the two matrix operations needed to complete the algorithm

function ret = fibmat(n)
  A = [1,1; 1,0];


endfunction

%!assert (fibmat(0) == 0);
%!assert (fibmat(1) == 1);
%!assert (fibmat(2) == 1);
%!assert (fibmat(3) == 2);
%!assert (fibmat(4) == 3);
%!assert (fibmat(5) == 5);
%!assert (fibmat(6) == 8);
%!assert (fibmat(25) == 75025);

Performance comparison

Time: 10 minutes
Activity: Demo / discussion

We can expect the second Fibonacci implementation to be faster for two distinct reasons

It's possible to compute matrix powers rather quickly (O(log n) compared O(n)), and since the fast algorithm is also simple, we can hope that octave implements it. Since the source to octave is available, we could actually check this.
Octave is interpreted, so loops are generally slower than matrix operations (which can be done in a single call to an optimized library). This general strategy is called vectorization, and applies in a variety of languages, usually for numerical computations. In particular most PC hardware supports some kind of hardware vector facility.

Of course, the first rule of performance tuning is to carefully test any proposed improvement. The following code gives an extensible way to run simple timing tests, in a manner analogous to the Python timeit method, whose name it borrows.

# Based on an example from the Julia microbenchmark suite.

function timeit(func, argument, reps)
    times = zeros(reps, 1);

    for i=1:reps
      tic(); func(argument); times(i) = toc();
    end

    times = sort(times);
    fprintf ('%s\tmedian=%.3fms mean=%.3fms total=%.3fms\n',func2str(func), median(times)*1000,
             mean(times)*1000, sum(times)*1000);
endfunction

What are the new features of octave used in this sample code?

tic, toc, from GOBG8, GO 36.1
Function Handles
what else?

We can either use timeit from the octave command line, or build a little utility function like

function bench
  timeit(@fib, 42, 100000)
  timeit(@fibmat, 42, 100000)
endfunction

Overview

This assignment is based on the material covered in Lab 14 and (particularly) Lab 15.

The goal of the assignment is to develop a simple query language that lets the user select rows and columns from a CSV File, in effect treating it like database.

General Instructions

Every non-test function should have a docstring
Feel free to add docstrings for tests if you think they need explanation
Use list and dictionary comprehensions as much as reasonable.
Your code should pass all of the given tests, plus some of your own with different data. If you want, you can use some of the sample data from the US Government College Scorecard. I've selected some of the data into smaller files:

2013-1000.csv.gz
Sun 30 Oct 2022 11:22:46 AM

2014-1000.csv.gz
Sun 30 Oct 2022 11:22:46 AM

2014-100.csv.gz
Sun 30 Oct 2022 11:22:46 AM

2013-100.csv.gz
Sun 30 Oct 2022 11:22:46 AM

2015-1000.csv.gz
Sun 30 Oct 2022 11:22:46 AM

2015-100.csv.gz
Sun 30 Oct 2022 11:22:46 AM

A marking rubric is available.

Reading CSV Files

We will use the builtin Python CSV module to read CSV files.

def read_csv(filename):
    '''Read a CSV file, return list of rows'''

    import csv
    with open(filename,'rt',newline='') as f:
        reader = csv.reader(f, skipinitialspace=True)
        return [ row for row in reader ]

Save the following as "~/fcshome/assignments/A5/test1.csv"; we will use it several tests. You should also construct your own example CSV files and corresponding tests.

name,   age,    eye colour
Bob,    5,      blue
Mary,   27,     brown
Vij,    54,     green

Here is a test to give you the idea of the returned data structure from read_csv.

def test_read_csv():
    assert read_csv('test1.csv') == [['name', 'age', 'eye colour'],
                                     ['Bob', '5', 'blue'],
                                     ['Mary', '27', 'brown'],
                                     ['Vij', '54', 'green']]

Parsing Headers

The first row most in most CSV files consists of column labels. We will use this to help the user access columns by name rather than by counting columns.

Write a function header_map that builds a dictionary from labels to column numbers.

table = read_csv('test1.csv')

def test_header_map_1():
    hmap = header_map(table[0])
    assert hmap == { 'name': 0, 'age': 1, 'eye colour': 2 }

Selecting columns

Use your implimentation of header_map to write a function select that creates a new table with some of the columns of the given table.

def test_select_1():
    assert select(table,{'name','eye colour'}) == [['name', 'eye colour'],
                                                   ['Bob',  'blue'],
                                                   ['Mary', 'brown'],
                                                   ['Vij',  'green']]

Transforming rows into dictionaries

Sometimes it's more convenient to work with rows of the table as dictionaries, rather than passing around the map of column labels everwhere. Write a function row2dict that takes the output from header_map, and a row, and returns a dictionary representing that row (column order is lost here, but that will be ok in our application).

def test_row2dict():
    hmap = header_map(table[0])
    assert row2dict(hmap, table[1]) == {'name': 'Bob', 'age': '5', 'eye colour': 'blue'}

Matching rows

We are going to write a simple query languge where each query is a 3-tuple (left, op, right), and op is one of ==, <=, and >=. In the initial version, left and right are numbers or strings. Strings are interpreted as follows: if they are column labels, retrieve the value in that column; otherwise treat it as a literal string. With this in mind, write a function check_row that takes a row in dictionary form, and checks if it matches a query tuple.

def test_check_row():
    row = {'name': 'Bob', 'age': '5', 'eye colour': 'blue'}
    assert check_row(row, ('age', '==', 5))
    assert not check_row(row, ('eye colour', '==', 5))
    assert check_row(row, ('eye colour', '==', 'blue'))
    assert check_row(row, ('age', '>=', 4))
    assert check_row(row, ('age', '<=', 1000))

Extending the query language

Extend check_row so that it supports operations AND and OR. For these cases both left and right operands must be queries. Hint: this should only be a few more lines of code.

def test_check_row_logical():
    row = {'name': 'Bob', 'age': '5', 'eye colour': 'blue'}
    assert check_row(row, (('age', '==', 5),'OR',('eye colour', '==', 5)))
    assert not check_row(row, (('age', '==', 5),'AND',('eye colour', '==', 5)))

Filtering tables

Use you previously developed functions to impliment a function filter_table that selects certain rows of the table according to a query.

def test_filter_table1():
    assert filter_table(table,('age', '>=', 0)) == [['name', 'age', 'eye colour'],
                                                    ['Bob', '5', 'blue'],
                                                    ['Mary', '27', 'brown'],
                                                    ['Vij', '54', 'green']]

    assert filter_table(table,('age', '<=', 27)) == [['name', 'age', 'eye colour'],
                                                     ['Bob', '5', 'blue'],
                                                     ['Mary', '27', 'brown']]

    assert filter_table(table,('eye colour', '==', 'brown')) == [['name', 'age', 'eye colour'],
                                                                 ['Mary', '27', 'brown']]

    assert filter_table(table,('name', '==', 'Vij')) == [['name', 'age', 'eye colour'],
                                                         ['Vij', '54', 'green']]

def test_filter_table2():
    assert filter_table(table,(('age', '>=', 0),'AND',('age','>=','27'))) == [['name', 'age', 'eye colour'],
                                                                              ['Mary', '27', 'brown'],
                                                                              ['Vij', '54', 'green']]


    assert filter_table(table,(('age', '<=', 27),'AND',('age','>=','27'))) == [['name', 'age', 'eye colour'],
                                                                               ['Mary', '27', 'brown']]

    assert filter_table(table,(('eye colour', '==', 'brown'),
                               'OR',
                               ('name','==','Vij'))) == [['name', 'age', 'eye colour'],
                                                        ['Mary', '27', 'brown'],
                                                        ['Vij', '54', 'green']]

Posted Fri 18 Nov 2022 04:30:00 PM Tags: /tags/assignment /tags/cs2613 /tags/python

Python Background

Octave Background

Before the lab

Counter, generators, and classes

Fibonacci, again

Running Octave

Fibonacci

Fibonaccci as matrix multiplication

Performance comparison

Overview

General Instructions

Reading CSV Files

Parsing Headers

Selecting columns

Transforming rows into dictionaries

Matching rows

Extending the query language

Filtering tables

Before the lab

Background

Discussion

Regular Expressions

Stripping Quotes

Handling quoted commas

Parsing more columns

Before the Lab

Background

Discussion

Globbing and List comprehensions

Dictionary Comprehensions

Filtered List Comprehensions

Using format

Before the Lab

Discussion

A first example

Pytest

Modules

More testing, docstrings

Indentation

Exceptions

On your own

Introduction

Hack

Conclusions

Using `format`