Saurik: What to And What Not to Expect From the Next Version of Cydia

In a recent post about the next version of Cydia, we got some great feedback from users about what features they’d like to see in Cydia 1.1.

The creator of Cydia, Jay “saurik” Freeman posted a long and informative response to our article that gave some insight into the current and future state of Cydia. He chose many questions posed in the article’s comments to address key issues.
We’re very grateful that saurik decided to share this information with us. He addresses several key topics concerning Cydia…

Features of Cydia 1.1 will include:

• the ability to run and operate Activator, libstatusbar, and SimulatedKeyEvents while Cydia is running
• an overall speed improvement, including the “Loading Changes” dialog
• “much less” memory usage
• a more advanced search mechanism with a new relevancy algorithm
• better management of broken repositories

Here’s everything saurik had to tell us about Cydia:

Multitasking

“This is simply not possible at this time. I realize everyone wants it, and hell: I want it, too; but everyone saying it is important doesn’t make it possible. The reason Cydia doesn’t have iOS 4 compliant multi-tasking is that, to make the modifications it makes to the system, it runs as “root”, a user that has more permissions on the system than anything else, which means that SpringBoard, a lowly process running as “mobile”, cannot suspend/resume it.
Now, this is something that /can/ be remedied, and is something that I’ve been thinking of how to do for a long time, but all the obvious ways people like to bring up for making Cydia’s GUI run as mobile with only small parts running as root would make Cydia run slower, and speed is something that is primary on everyone’s minds when they are using Cydia. Luckily, there are things I’ve figured out that may make this more reasonably possible, but certainly not for this release.”

Leaving Mobile Substrate On When Cydia is Open

“If you did this you would find your system would suddenly become unusable. I realize that it sounds all nice and fuzzy that Mobile Substrate should modify all applications on the system, including Cydia, but again: Cydia runs as root. Almost all extensions in the ecosystem are NOT designed with this in mind, and when suddenly given root access start destroying the permissions of your configuration files and Media folders, making all normal applications unable to use them.
Therefore, with this release of Cydia, I’ve gone through the “big ticket items”–which seem to come down to Activator (what starts SBSettings), libstatusbar (adds notification items to the status bar), and SimulatedKeyEvents (injection of key events from Veency)–to verify with their developers that they will work correctly in an environment running as root. These extensions (plus WinterBoard, which doesn’t work on root on 4.x but is harmless, and will be fixed in a future release) are what are available from inside of Cydia until Cydia is modified to run as mobile.”

Better Looking Interface and Backup Option

“A backup function actually does not require pushing a new build of Cydia, but it requires time to figure out how to scale the users to support the kind of load Cydia has. Cydia is running with many orders of magnitudes more users than any of its competitors have, which means that a lot of things that people like to think “should be simple, X did it” are actually much harder to implement. I also keep privacy at the forefront of my mind while building features like this, and want to be 100% certain that no one can get access to your installed products list other than you.
As for a “better looking interface”, I try hard to maintain something that competes with Apple’s products. A few things rotted on 4.x (the positions and sizes of some buttons), and the various “black” interfaces (the black bar and the black screen) get mixed reports, but otherwise the main problem users have with Cydia is not Cydia: it is with repositories. Every time I’ve gotten actual feedback “this specific thing is bad”, where that thing isn’t something that Apple themselves do in their iTunes or App Store applications (which should be taken as the “intuitive model”), it is in areas of the interface I simply do not have any control over: the content shown for a package by the repository.”

Confusion With the Term “Changes”

“Maybe I’m crazy, but I always thought of the word “Changes” to be a very non-geeky end-user term for “stuff that changed”. It certainly isn’t a technical term: it was not chosen because of some geeky desire to have the codebase match the UI, nor was it chosen because it had some esoteric meaning in Latin or Greek. It was instead chosen as it was a single word that immediately meant to the largest number of people I talked to exactly what that page did: showed you what changed. Regardless, “New Releases/Updates” certainly won’t fit on a tab label.”

Speed

“As with /every/ release of Cydia, Cydia 1.1 is faster than previous releases. In specific, it is faster than 1.0.3366 by a good margin, which itself was faster than 1.0.3222 by an even larger chasm. On this note, however, it is important to note that Cydia is tackling a hard problem: no other application I have seen on the iPhone, from Apple or any third party, is attempting to search index and manage tens of thousands of data items, on the client, in real time, aggregated from user-selected sources.
In contrast, Cydia has some of the fastest technology in existence with regards to handling this data, whether it be custom algorithms (Cydia includes a locale-aware string comparison radix sort, which AFAIK is the fastest sorting algorithm in any iOS application) or special on-disk data structures (new in 1.1 is “Cytore”, a new way to store local metadata on packages that can be loaded nearly instantaneously from flash; for those out there who are technically minded, it is an on-disk memory mapped hashtable, which drastically beats out alternatives people like to try to bring up such as SQLite).”

Loading Times

“Despite myths to the contrary, the amount of data displayed in the Changes list does not drastically affect how quickly it loads. There /was/ a bug in many versions of Cydia 1.0 that caused there to be at least a little delay related to the number of items on the list, but this bug was already fixed as of 1.0.3366. The cost of the calculation is deciding what entries should be on the list at all (and specifically which ones are actually updates vs. new releases), not displaying them all at once. That said, Cydia 1.0.3366 moves the loading of changes until after you click the tab, which makes it more evidence how much time is being spent on this feature (which itself is, again, faster on 1.1).”

Memory Usage

“Despite Cydia 1.1 continuing to attempt to juggle tens of thousands of items in memory, thanks to Cytore, it uses much less memory than ever before. Other optimizations have been made, as with every version of Cydia, in order to decrease the memory usage of the app as a whole. Additionally, and in particular, Cydia 1.1 is much more conscientious of memory warnings, and attempts to throw out as much state as possible during these events.
That said, the amount of memory on even reasonably modern devices (anything past the iPhone 3G) available for running applications (not in total, but available after Apple’s system applications get their share), is an order of magnitude greater: whereas on an iPhone 3G you were working with maybe 20MB of available memory, on an iPhone 3G[S] you have 150MB, and an on iPhone 4 you have 400MB available. So, despite Cydia 1.1 actually needing less memory to operate than Cydia 1.0, the pressure on memory is pretty much gone, and will not affect future users thanks to hardware upgrades.”

Advanced Search

“Unfortunately, this device is simply too slow to provide “advanced search capabilities”, and certainly not suggestions, given the constraints of “from user-selected repositories” “in something resembling real time”. That said, Cydia 1.1 has a much better search mechanism, including an integer-arithmetic radix-sorted relevancy algorithm I managed to implement.
What would really be needed to have a truly amazing search experience is to not do searching on the client: to instead handle it on my servers. This is how products like the App Store, Kindle, or Netflix work: it is not at all common for services users are used to to attempt to manage the entire database /on the device/, doing local searching, rather than having the data and computation for that existing in offline-indexed search structures on a massive server in the cloud.
Unfortunately, the reason people use Cydia are varied, and many people are using Cydia with repositories that frankly they shouldn’t be: whether the repository contains software that is dangerous (a niche community with tweaks receiving minimal testing, or using bad practices like on-disk file patching) or downright illegal (there are things you are allowed to do in your country that I cannot in mine), I am certainly not going to be acting as the centralized storage and indexing gateway for people to find and manage this content.
Instead, what keeps people coming back to Cydia is the fact that it acts as the fundamental alternative: that rather going to Apple, with their carefully curated set of centralized experiences, you go to Cydia, “the wild west of software”, where software modifies other software in a kind of reckless abandon that is going to lead to pain even in the best possible scenarios, and in the worst possible worlds is going to lead to things that you will not be able to list on a default repository, and which Cydia may even warn you about installing, but which you should still be able to access and even search for using Cydia’s search mechanisms.”

Error Messages

“Errors from Cydia do not come from Cydia. If you type a URL into Cydia for a broken repository, that repository is going to be low-quality and is going to cause you problems. If it is offline, Cydia is going to tell you it is offline, and if it is malformed Cydia is going to get angry about that. Cydia is simply going to sit there idly while there are a ton of broken and offline repositories in your list: it will tell you all of the errors involved in the hope that you will remove the broken repositories and get on with your life (which is a very apt metaphor, as most third party repositories are very slow, and cause your refresh experience to take a very very very long time).”

“Rate” and “Review” Sections

“We actually tried this, and it was a miserable failure: more time had to be spent moderating the reviews, most of which were misleading, inflammatory, or downright inappropriate, than anyone got value out of this mechanism: it was even worse than on the App Store, which is notorious for bad reviews (people often rate down a package for inane reasons, making the data horribly invalid).
Given these issues, I attempted to put together a vision of how comments and ratings could work in Cydia, and even made a trial implementation (screenshots were even handed out at some points, and I did demos at a few conferences), but when word came up that I was even considering releasing it, I received strong pushback from some of the best developers in the ecosystem–the people you are most likely to want to give mega-good reviews to–that if I continued with that they would give up on the ecosystem, due to the issues from before.
And, to be honest, I am not certain that I would have solved those problems, and given subsequent experiences from alternative products, and looking at how people used the ratings, what people said in the comments, and how things finally got rated, I no longer believe that I would have: I believe the concept of the off-the-shelf “comments and ratings” to be a fundamentally flawed system that inherently leads to abuse.
Now, not all rating systems need to be “off-the-shelf”, so something truly innovating and “actually solving the problem” is what I hope to one day provide for Cydia. In the meantime, however, I always do my best to avoid injecting seriously suboptimal tradeoffs into our ecosystem.”

Improved Compatibility Listings

“Cydia has, for a while, contained numerous features that would allow repositories to help with this problem.

1. a mechanism to specify firmware compatibility on packages (packages can Depends: specific firmware revisions).
2. the Cydia Store lets vendors block purchases for specific firmwares (any paid product can register its compatibility with its repository, and then I will filter it to users who can use it).
3. the firmware version is sent as part of the user-agent to the web pages for each product, allowing developers to display their own warnings.
4. compatibility is even more specially able to be done by feature detection, allowing packages to say “I need voiceover support on a device with a camera running an armv7 CPU and a retina screen”.

In essence, there is very little excuse for packages, repositories, products, or anything else in the Cydia ecosystem to be poorly specified in terms of firmware compatibility. That said, almost no packages in the ecosystem, and even very few products (where one would imagine this to be the most important), have this information included at any of these levels, which is rather disappointing.
So, Cydia 1.1 is not going to attempt to improve on any of these mechanisms, as Cydia 1.0 already has more than enough of them: the real onus is now on the developers and artists of specific items.”

Removal of Dead Content

“I do not have any control over what content is available in Cydia. I mean, I can refuse to personally accept money for it, but I have almost no introspection over things that are either free or sold on the developer’s website. For years I have attempted to get repositories to pull obsolete packages: they refuse. Instead of lobbying me, who agrees with you and is powerless, you need to be sending these complaints to the default repositories: BigBoss, ModMyi, and ZodTTD.”

Clarified Install Requirements

(Tags saying if installation of apps/tweaks, etc. need a springboard refresh or if the device needs to reboot)

“While this is often stated, this is simply not how this mechanism works: packages compute whether they need a reboot or reload as they install, allowing packages to make optimizations like “I only need to reboot if the user is using this firmware version and has this other package installed with this setting specified”. In fact, all of my packages that need features like this attempt these optimizations, and often you will not need as many reboots or reloads because of it.
Therefore, specifying this as static tags on a package would increase the number of reboots a user has to perform needlessly. That said, for packages where it is not obvious (extensions are going to require a reload, and MobileSubstrate is going to require a reboot), such as cases of MMS clients that require a reboot, it should certainly be best practice for the developer to put this information on their package information screen. This is even easier for the developer/vendor than modifying the package, and even then is very uncommonly specified: adding the tag therefore won’t even change how often it is reported.”

Repo Management

“With regards to repositories that did not add correctly, as stated by Ryan Petrich, Cydia 1.1 should no longer end up in situations where broken repositories are so unusable that they are also undeletable. That said, many users complain about repositories installed via a package: to delete these repositories you will need to remove the package that represents them.
(Due to some of these complexities, it is Cydia policy going forward that no repositories will be installable from default repositories via packages, and the existing ones under More Sources will be transitioned to a new mechanism for handling these that has been added that will allow more direct, simpler, and safer manipulation of repositories using a soon-to-be-revamped More Sources page.)”

Saurik’s closing comment,

“Thank you all, by the way, for your interest in Cydia: the fact that you care at all about what features are or are not in Cydia 1.1 means a lot to everyone working on the project.”

We want to sincerely thank saurik for his detailed and informative responses to our reader’s questions. We wish he and his team the very best in making Cydia the best it can be!
Hopefully you learned a few things from this lengthy discussion. Feel free to discuss or share opinions about what saurik had to say in the comments below.