TDDB68
Lesson 3
Simon Ståhlberg
Contents
•
•
Overview of lab 3
• A: Exec, wait, exit
•
B: Program arguments
•
C: Termination of ill-behaving user processes
•
Testing your implementation
Overview of lab 4
• File system
•
The readers-writers problem
•
Additional system calls
•
Testing your implementation
Lab 3: Overview
•
Part A: Execution of several user processes
• Implement Exec, Wait and Exit
• Exec: starts up a child process that executes the given file
•
•
•
•
•
Regardless of the order of wait and exit, wait must return the child’s exit status. We take a
closer look at some scenarios that you have to think about
Alive counting to decide who should free dynamically allocated memory
Part B: Handling program arguments
• Currently, Pintos does not support arguments to programs
•
•
•
Exit: terminates the process, frees allocated resources and saves the exit status somewhere
Wait: waits on a child process to terminate and return its exit code
Remember *esp = PHYSBASE - 12? We needed - 12 (bytes) to compensate for the missing
arguments. Remove it and add proper support
What is on the initial stack
Part C: Termination of ill-behaving user processes
• How to be paranoid about user arguments
•
Stack pointer, pointers in general, strings, and buffers
Lab 3: Part A - Exec
•
pid_t exec (const char *cmd_line)
•
The first word of the string cmd_line is a file name, the rest are the
arguments to the program. Spawn a new child process that loads the
file and executes it. If the child process could load and start executing
the file then return the process ID (PID) of the child, return -1 otherwise
•
The current implementation of exec does not wait to see if the child
could load the file/program
•
Make exec wait for the new process to finish loading before returning
the PID
•
Hint: The “current” implementation is process_execute
•
Hint: You can assume that TID and PID are the same thing in Pintos
•
Hint: A process can have several children at the same time
Lab 3: Part A - Exec
s
s
e
p
_
rt
ta
s
c
o
r
d
oa
l
e
h
(t
d
e
d
a
lo
Parent
re
at
e
th
re
ad
_c
s_
es
oc
pr
(s
om
e
co
de
)
ex
ec
u
te
Child
d
l
i
h g
c
tin
e
u
h
t ec
r
fo ex
t
i
a art
W st
to
Wa
an ke u
wh d no p th
ex ethe tify t e pa
ec
h
uti r it c e pa rent
ng an
r
sta ent
rt o
r
p
)
m
a
r
g
Lab 3: Part A - Exec
•
The following functions and lines of code are of interest.
•
tid_t process_execute (const char *file_name)
•
tid = thread_create (file_name, PRI_DEFAULT,
start_process, fn_copy); // Inside process_execute
•
tid_t thread_create (const char *name, int
priority,
thread_func *function,
void *aux)
•
static void start_process (void *fname)
•
start_process
is the function that the new thread starts
executing, and aux is the argument to said function
Lab 3: Part A - Exec
•
Hint: The only place where start_process is
"called" is in process_execute. Hence, you can
extend the parameter of start_process (fname) and
the argument to thread_create (fn_copy) to
whatever you want (e.g. a pointer to a struct)
•
Hint: The initial thread “main” does not have any
parent, and is not created by thread_create, but it
is initialised by init_thread
Lab 3: Part A - Exit
•
void exit (int status)
•
The exit status has to be made available to its parent process. More
on this shortly
•
The process has to release all its allocated resources, such as free
dynamically allocated memory, close opened files, close network
connections, and so on. Release whatever resource you allocate in
your solution
•
If the process crashes for any reason, then the exit status should be
-1. Furthermore, all of the allocated resources must be released
•
Hint: Release resources in thread_exit() or process_exit()
•
Hint: At exit, do: (make check will fail otherwise)
printf("%s: exit(%d)\n", thread-name, thread-exit-value)
Lab 3: Part A - Wait
•
int wait (pid_t pid)
•
The calling process waits for the child process with the PID to terminate and
receives its exit status. If the child has already terminated, its exit status should
be returned immediately
•
In the last case, the exit status must be saved somewhere since the process
does not exists
•
Scenarios:
• Parent calls wait before the child terminates
• Parent calls wait after the child terminates
• Parent terminates before the child without calling wait
• Parent terminates after the child without calling wait
•
In each of these scenarios, your code must work and shared resources must
be released by whoever terminates last
•
Remember that a process can have several children!
Lab 3: Part A - Wait
•
The exit status is only available until the first wait call by the
parent. If the parent waits twice on the same child, then the
second wait returns -1
•
Hint: Since the exit status has to be available even after the child
terminates, then the exit status can be stored in dynamically
allocated memory
•
Hint: A common approach is to have a struct of data for every
parent-child pair. The struct contains, amongst other things, the
exit status
•
As we noted earlier, this struct must be freed by whoever
terminates last, i.e. either the parent or the child
Lab 3: Part A - Wait
•
Scenario: parent waits for child to exit
•
Important! Busy waiting is NOT allowed!
exit(X)
Child
Parent
Free the
exit value!
Waits
exec
wait(Child)
returns X
Lab 3: Part A - Wait
•
Scenario: child exits before the parent, and then
the parent calls wait
exit(X)
Child
Parent
store X on the heap
exec
Free the
exit value!
wait(Child)
returns X
Lab 3: Part A - Wait
•
Scenario: parent never waits for the child and exits
before it
exit(X)
Child
Free the
exit value!
Parent
exec
exit(Y)
Lab 3: Part A - Wait
•
How to detect who is the last to exit?
•
Hint: Keep a counter, together with the exit status, with the
initial value 2. When the parent and child exit, decrement it by
1. The value 0 represents that both the parent and the child has
exited, and whoever decrements it to 0 should free the memory
•
Hint: Protect the counter with synchronisation! Pintos might
leak memory otherwise!
•
Child
struct parent_child {
int exit_status;
pcs
int alive_count;
/* ... whatever else you need ... */
};
Parent
Lab 3: Part B
•
Pintos does not currently support program
arguments. For example: insult -s 17
•
Remember *esp = PHYSBASE - 12? The arguments
require at least 12 bytes on the stack. Remove the
subtraction!
•
We will go through how to setup the stack in order
to support program arguments
Lab 3: Part B
•
To run a program with arguments in Pintos:
•
Pintos (note the single quotes):
pintos --qemu --run 'insult -s 17'
•
System call: exec("insult -s 17");
Lab 3: Part B
•
Suppose we do this: insult -s 17
•
The parameters of the main function of C programs are int argc
and char **argv. In this example:
•
is 3
•
argc
•
argv[0]
is "insult\0"
•
argv[1]
is "-s\0"
•
argv[2]
is "17\0"
•
argv[argc]
is NULL
Note that the length of the array is actually 4, but the last element
is always NULL (required by the C standard)
Lab 3: Part B
•
Every time you do a function call, a stack frame is
pushed on the stack:
Parameters
Stack frame
{
Return address
Growth
direction
Local variables
•
The function main is never really called, but the layout
is the same
•
The parameters and the return address of the stack
frame is pushed onto the stack by Pintos (your code!)
•
We touched on this in the first lesson
{
{
Lo
ca
l va
{
s
ria
ble
Pa
ram
ete
rs
Th
es
trin
on g i
the s st
sta ored
ck
Lab 3: Part B
{
Address
Name
Data
Type
0xbffffffd
0xbffffffa
0xbffffff3
...
0xbffffff0
0xbfffffec
0xbfffffe8
0xbfffffe4
0xbfffffe0
...
0xbfffffc8
0xbfffffc4
0xbfffffc0
argv[2][...]
argv[1][...]
argv[0][...]
word-align
word-align
argv[3]
argv[2]
argv[1]
argv[0]
"17\0"
"-s\0"
"insult\0"
unused
unused
NULL
0xbffffffd
0xbffffffa
0xbffffff3
char[3]
char[3]
char[7]
Stack grows
towards 0
argv
0xbfffffe0
argc
3
return address
unused
Ad
po just
di inte the
vis r
ib so stac
le it
by is k 4 }
char
char
char
char
*
*
*
*
char **
int
void (*) ()
Adding all of this to the stack is
what Lab 3: Part B is about!
}
Th
ne e f
ve un
r r cti
et on
ur
ns
Lab 3: Part B
•
Step 1: Change *esp = PHYSBASE - 12; to *esp = PHYSBASE; (the size of the argc (int), argv (char**) and argv[0] is 12)
•
Step 2: Put the words on the stack, word alignment, the argv array,
argc and the return address
•
Hint: start_process calls load, which calls setup_stack.
•
Hint: Take a look at setup_stack(void **esp). Note that the
argument is a pointer to the stack pointer! In other words, you must
dereference twice to write to the stack, and dereference once to
change the stack pointer itself!
•
Hint: setup_stack is called by load, and after the call there is
debug code for printing the stack. To print the stack, uncomment /*#define STACK_DEBUG*/
Lab 3: Part B
•
Hint: You get a string, such as "insult -s 17\0", and you need to split it
up in smaller parts. Use (in lib/string.[c|h]): char * strtok_r(char *s, const char *delimiters, char **save_ptr)
•
The following loop tokenises the string s = "insult -s 17\0". for (token = strtok_r (s, " ", &save_ptr); token != NULL;
token = strtok_r (NULL, " ", &save_ptr)) { ... }
Every iteration you get the next word, i.e. "insult\0", "-s\0" and
"17\0". However, it does so destructively!
•
•
This is what the memory looks after the loop: s = "insult\0-s\017\0"
•
You need to save a pointer to every word
Hint: Read Pintos documentation 3.5.1, it presents another stack print!
Lab 3: Part C
•
Argument paranoia: nothing the user processes does should be
able to crash Pintos
•
All user processes i Pintos, even when running in unsupervised
mode, use virtual memory (a page table). Dereferencing a pointer
not in the page table leads to errors such as invalid page fault or
segmentation fault
•
For example: create((char*) 1, 1); is an invalid call since
address 1 is in kernel space. Furthermore, it is not a string!
•
Another example: read(STDIN_FILENO, 0xc0000000, 666); writes
data to kernel space (will likely overwrite something important)
•
Hence, ALL pointers from the user processes to the kernel must
be checked! Including the stack pointer, strings, and buffers
Lab 3: Part C
How to validate a pointer:
•
A valid pointer from a user process is:
•
below PHYS_SPACE in virtual memory (not kernel memory)
•
associated with a page in the page table for the process (use
pagedir_get_page)
•
If some pointer is not valid, then terminate the process! The exit status
is then -1
•
If you were to dereference an invalid pointer that is below PHYS_SPACE,
then you get an error. It is possible to take care of the problem when
the error occurs, but it is more challenging (although, it is faster: read
the documentation if you want to attempt this)
•
Hint: Read Pintos documentation 3.1.5
Lab 3: Part C
•
Suppose a process calls create((char *) PHYS_BASE 12345, 17);
•
The function filesys_create does not check the string,
and the string is not NULL. The call will likely crash Pintos!
•
Hint: You must check that the char * is a string by
iterating over every character of it, and check that the
pointer to it is a valid pointer
•
Hint: In other words, for strings, you must check every
character until you find a '\0'. Remember to first check the
pointer, then read it! (You might get an error otherwise)
Lab 3: Part C
•
Suppose a process calls write(1, malloc(1), 1000000);
Obviously, the arguments are invalid since the size of the buffer is
1, but the process claims it is much bigger!
•
If we were to read from the buffer, then we would get an error. It is
not sufficient to only test the pointer, we must also check its size!
•
Hint: We must check that every possible pointer to the buffer is
valid. In other words, we must test 10000000 pointers (at most, if
you want to optimise it then you can compute where the page
boundaries are, and check those)
•
Hint: In contrast to strings, the size of the buffer is given and we
do NOT have to search for a '\0'
Lab 3: Part C
•
The user process can modify its own stack pointer:
asm volatile (”movl $0x0, %esp; int $0x30” ::: );
•
So you have to check the stack pointer if you want
to read what it points to. If you increment the stack
pointer, then you have to redo the check!
•
In other words, you have to check the stack pointer
before reading the system call number, before
reading the first argument, and so on
Lab 3: Testing
•
Sadly, you cannot test until you have finished both part A and part B
•
When you have finished part A and B, then you can test your
implementation with: make check
•
The following tests are for Lab 1: halt,
•
Most of the exec-* and wait-* tests (and Lab 1) should pass when
you have finished Part A and B
•
Hint: You can run a single test (from userprog/build) with:
exit, create-normal,
create-empty, create-null, create-long, create-exists, createbound, open-normal, open-missing, open-boundary, open-empty,
open-twice, close-normal, close-stdin, close-stdout, close-badfd, read-boundary, read-zero, read-stdout, read-bad-fd, writenormal, write-boundary, write-zero, write-stdin, write-bad-fd
make tests/userprog/halt.result
•
The rests are for when Part C is finished (~60 tests in total)
Lab 4: Overview
•
Synchronising the file system in Pintos
•
The readers-writers problem
•
Additional system calls (seek, tell, filesize, remove
•
Testing your implementation
•
Lab 3 is time consuming, but Lab 4 should take
about as much time as Lab 1
Lab 4: File system
•
devices/disk.[h|c] - Low-level operations on the drive (shared and already synchronised)
•
filesys/free-map.[h|c] - Operations on the map of free disk sectors
(shared)
•
filesys/inode.[h|c] - Operations on inodes, which represents an
individual file. When you write/read data to/from an inode then you
modify the actual physical file (shared)
•
filesys/filesys.[h|c] - Operations on the file system, such as
create, open, close, remove, and so on (shared)
•
filesys/directory.[h|c] - Operations on directories (partially shared)
•
filesys/file.[h|c] - A file object that contains an inode and things
like a seek position. Every process has its own object (not shared)
Lab 4: File system
•
Your assignment is to synchronise the files that
contains data which is shared between multiple
processes and are not already synchronised
•
Use locks and semaphores!
Lab 4: Readers-writers
•
It is easy to synchronise the file system by only using one lock
everywhere, but that leads to extremely poor performance
•
We have these requirements:
•
•
Several readers are able to read from the same file at the same time
•
Only one writer is able to a specific file at the same time
•
Several writer are able to write to different files at the same time
•
When a process is reading from a file, then no process can write to
the file at the same time
•
When a process is writing to a file, then no process can read from
the file at the same time
Hint: There is at most 1 inode per physical file
Lab 4: Readers-writers
•
The readers-writers problem is how to achieve the
aforementioned requirements. There are several
solutions with different properties
•
Hint: Wikipedia has a few algorithms https://en.wikipedia.org/wiki/Readers–writers_problem
•
Hint: The solution with readers-preference is easy
to implement, but might starve writers
Lab 4: System calls
•
In addition to synchronising the file system, you also have to implement
these system calls:
• void seek (int fd, unsigned position): Sets the current seek position of the file
• unsigned tell (int fd):
Gets the seek position of the file
•
int filesize (int fd):
Returns the file size of the file
•
bool remove (const char *file_name):
Removes the file. Hint: The removal of the file is delayed until it is not
opened by any process (make sure that this counter is synchronised)
•
Hint: Use built-in functions
•
Hint: Check for invalid arguments!
Lab 4: Hints
•
Some questions that you should ask yourself: What can happen, in
the worst case, if two processes try to...
•
Create and remove the same file at the same time?
•
Create and remove the same directory at the same time?
•
Open the same file at the same time?
•
Open and close the same file at the same time?
•
And so on!
•
Many of these operations should, from each other perspective, be
“atomic"
•
This is NOT a complete list!
Lab 4: Testing
•
•
To test your implementation, we provide the following user programs:
•
pfs.c
•
pfs_reader.c
•
pfs_writer.c
The program pfs read and write to the same file concurrently:
•
2 writers that repeatedly fill the file with an arbitrary letter
•
3 readers that repeatedly read the file and checks that all the letters are the same
•
If the letters differ for the readers, then the test fails
•
Run pfs many times and check the result each time! Inconsistency due to lack of
synchronisation may not happen every time
•
Hint: The Linux Mint computers might be too fast, change #define TIME_SLICE 4
(threads/thread.c) to a lower value, or #define TIMER_FREQ 100 (devices/
timer.h) to a higher value, to make the scheduler to switch between processes more
often
Questions?
Remember!
If you pass the labs by
March 10, 2017 in webreg then you get bonus points!