Update the docs for the directory structure changes (#813)

* Update the docs for the directory structure changes

docs/src/SUMMARY.md

@@ -18,9 +18,10 @@
 
 - [Developer Documentation](./developer/introduction.md)
   - [Basics](./developer/basics.md)
-  - [Unwritten Rule](./developer/unwritten_rules.md)
+  - [Unwritten Rules](./developer/unwritten_rules.md)
   - [Good places to start](./developer/good_places_to_start.md)
   - [This Documentation](./developer/documentation_mdbook.md)
+  - [Repository Structure](./developer/repo_structure.md)
   - [Crate Specific Information](./developer/crate_specific_information.md)
     - [libcgroups](./developer/libcgroups.md)
     - [libcontainer](./developer/libcontainer.md)

docs/src/developer/basics.md

@@ -8,6 +8,21 @@ Youki is a low level container runtime, which deals with the creation and manage
 
 Before you start working on developing youki, you should go through [the User documentation](../user/introduction) as it specifies the requirements and setup for running youki. For developing youki, you will need to install the dependencies and clone the repo, as specified in the [Basic Setup](../user/basic_setup.md) and [Basic Usage](../user/basic_usage.md) sections.
 
+## Testing while developing
+
+While developing youki, you might need to compile and test the code from time to time, to make sure it is working and and something is not accidentally broken. Currently there are two ways to verify that:
+
+- Unit tests, which test individual components of youki
+- Integration tests, which test the complete functionality of youki commands from start to end.
+
+As the steps to run these tests can be a bit tedious, a makefile in project the root provides an easy way to run these quickly. The makefile currently states three individual test :
+
+- test: The unit tests
+- oci-integration-test: The integration tests provided by OCI, these are the current standard to make sure youki is OCI compliant.
+- integration-test: This is the Rust port of the OCI runtime tests, as there are some issues in the OCI tests. See [integration_test](./integration_test.md) page.
+
+All three can be run by using `make test-all`, or you can run the individual command to run specific tests.
+
 ## Resources
 
 #### OCI

docs/src/developer/integration_test.md

@@ -1,5 +1,7 @@
 # integration_test
 
+**Note** that these tests resides in /tests/rust-integration-tests/integration_test at the time of writing.
+
 This crate contains the Rust port of OCI-runtime tools integration tests, which are used to test if the runtime works as per the OCI spec or not. Initially youki used the original implementation of these test provided in the OCI repository [here](https://github.com/opencontainers/runtime-tools/tree/master/validation). But those tests are written in Go, which made the developers depend on two language environments Rust and Go to compile youki and test it. The Validation tests themselves also have an optional dependency on node js to parse their output, which can make it a third language dependency.
 
 Other than that, those tests also showed some issues while running on some local systems, and thus running the tests would be difficult on local system. As the runtime is a complex piece of software, it becomes useful to have a set of tests that can be run with changes in code, so one can verify that change in one part of youki has not accidentally broken some other part of youki.

docs/src/developer/repo_structure.md

@@ -0,0 +1,27 @@
+# Repository Structure
+
+This page might be the one that gets most easily outdated, as the structure might change at any time! Thus make sure to update this whenever there are any changes in the overall structure of the whole repo. For the same reason, this does not list the structure in detail but instead describes only the main directories.
+
+### .github
+
+Contains workflows and files needed by those workflows.
+
+### crates
+
+This is the core of youki. This contains various libraries that are developed alongside of youki and the youki binary itself.
+
+### docs
+
+The directory where the source of this documentation resides. The source is also divided into two parts, for developers and users. Please see [Documentation documentation](./documentation_mdbook.md) for more information.
+
+### hack
+
+As the name suggests, contains hack scripts for patching some issues which are currently not solvable in a straightforward way or solving issues for which we have no idea of why they occur.
+
+### Scripts
+
+Contains scripts for various purposes, such as building youki, running integration tests etc. These might be small scripts called from many other scripts, big scripts that perform a complex task or helper scripts for the main makefile.
+
+### tests
+
+This contains all the integration tests for validating youki. Note that these are integration tests for start-to-end testing of youki commands. Unit tests for individual parts are in their respective source files in crates.

docs/src/developer/runtimetest.md

@@ -1,5 +1,7 @@
 # Runtime Test
 
+**Note** that these tests resides in /tests/rust-integration-tests/runtimetest at the time of writing.
+
 This crate provides a binary which is used by integration tests to verify that the restrictions and constraints applied to the container are upheld by the container process, from inside the container process. This runs the tests one-by-one, and the failing test prints the error to the stderr.
 
 ## Notes

docs/src/developer/test_framework.md

@@ -1,5 +1,7 @@
 # test_framework
 
+**Note** that these tests resides in /tests/rust-integration-tests/test_framework at the time of writing.
+
 This crate contains the testing framework specifically developed for porting the OCI integration test to rust. This contains structs to represent the individual tests, group of tests and a test manager that has responsibility to run tests. This Also exposes traits which can be used to implement custom test structs or test group structs if needed.
 
 By default the test groups are run in parallel using the [crossbeam crate](https://www.crates.io/crates/crossbeam), and the default test_group implementation also runs individual tests parallelly.

docs/src/developer/unwritten_rules.md

@@ -18,6 +18,10 @@ youki uses [log](https://crates.io/crates/log) crate to log information while ru
 
 Make sure that you comment copiously, and explain the peculiar behavior of your code so that others can understand why certain code is written the way it is. Also make sure to add doc comments and examples for `pub` items in the crates, so that users can find it from the docs generated by `cargo doc`.
 
+#### Scripts
+
+In any script, any makefile etc, make sure to `set -e` at the start. This will abort the script after any command fails, rather than continuing with incorrect state and creating knock-on errors.
+
 #### Update This Documentation
 
 Keep this Documentation updated! Make sure you add any relevant doc-links and resources to this that you found helpful or contains background information required to understand the code, so that it can help newcomers as well as others to find the resources in one single place.

docs/src/user/basic_setup.md

@@ -58,9 +58,8 @@ Once you have cloned the source, you can build it using
 ```console
 # go into the cloned directory
 cd youki
-cd scripts
+make build # or release-build
-# build
+./youki -h # get information about youki command
-./build.sh
 ```
 
 This will build the Youki binary, and put it at the root level of the cloned directory, that is in the youki/ .
@@ -106,7 +105,6 @@ VAGRANT_VAGRANTFILE=Vagrantfile.root vagrant up
 VAGRANT_VAGRANTFILE=Vagrantfile.root vagrant ssh
 
 # in virtual machine
-cd youki
+$ cd youki
-cd scripts
+$ make build # or release-build
-./build.sh
 ```

docs/src/user/introduction.md

@@ -11,3 +11,4 @@ This is divided into following sub-sections :
   - libcontainer
   - liboci-cli
   - libseccomp
+- Webassembly : This explains how to use webassembly module with youki.

scripts/README.md

@@ -1,5 +1,7 @@
 # Scripts
 
-This stores various scripts that do various things
+This stores various scripts that do various things. These can be intended to be used directly, or can be for using from some other scripts or Makefiles.
 
-TODO add details
+#### Note
+
+Please use `set -e` at the start of every script. This will ensure that the operation fails if any single command fails in that script. Without it, the script will continue after the failing command and might create a knock-on effect of incorrect results. In case you expect some step to fail, handle the failure directly rather than checking some condition to see if the command is successful in the rest of the script.