Is there a Destiny or is there only Free-Will?

You have quoted Cheiro’s successes, but what about his failures? I have looked at the book and was rather staggered by the number of prophecies that have failed to come off. You can’t deduce from a small number of predictions, however accurate, that all is predestined down to your putting the questions in the letter and my answer. It may be, but the evidence is not sufficient to prove it. What is evident is that there is an element of the predictable, predictable accurately and in detail as well as in large points, in the course of events. But that was already known; it leaves the question still unsolved whether all is predictable, whether destiny is the sole factor in existence or there are other factors also that can modify destiny,—or, destiny being given, there are not different sources or powers or planes of destiny and we can modify the one with which we started by calling in another destiny source, power or plane and making it active in our life. Metaphysical questions are not so simple that they can be trenchantly solved either in one sense or in another contradictory to it—that is the popular way of settling things, but it is quite summary and inconclusive. All is free-will or else all is destiny—it is not so simple as that. This question of free-will or determination is the most knotty of all metaphysical questions and nobody has been able to solve it—for a good reason that both destiny and will exist and even a free-will exists somewhere; the difficulty is only how to get at it and make it effective.
[SABCL, 22:467]

Fate and Karma

The Indian explanation of fate is Karma. We ourselves are our own fate through our actions, but the fate created by us binds us; for what we have sown, we must reap in this life or another. Still we are creating our fate for the future even while undergoing old fate from the past in the present. That gives a meaning to our will and action and does not, as European critics wrongly believe, constitute a rigid and sterilising fatalism. But again, our will and action can often annul or modify even the past Karma, it is only certain strong effects, called utkata karma, that are non-modifiable. Here too the achievement of the spiritual consciousness and life is supposed to annul or give the power to annul Karma. For we enter into union with the Will Divine, cosmic or transcendent, which can annul what it had sanctioned for certain conditions, new-create what it had created, the narrow fixed lines disappear, there is a more plastic freedom and wideness. Neither Karma nor Astrology therefore points to a rigid and for ever immutable fate.
[SABCL, 22:468]

Destiny and Personal Will

As for prophecy, I have never met or known of a prophet, however reputed, who was infallible. Some of their predictions come true to the letter, others do not,—they half-fulfil or misfire entirely. It does not follow that the power of prophecy is unreal or the accurate predictions can be all explained by probability, chance, coincidence. The nature and number of those that cannot is too great. The variability of fulfilment may be explained either by an imperfect power in the prophet sometimes active, sometimes failing or by the fact that things are predictable in part only, they are determined in part only or else by different factors or lines of power, different series of potentials and actuals. So long as one is in touch with one line, one predicts accurately, otherwise not—or if the lines of power change, one’s prophecy also goes off the rails. All the same, one may say, there must be, if things are predictable at all, some power or plane through which or on which all is foreseeable; if there is a divine Omniscience and Omnipotence, it must be so. Even then what is foreseen has to be worked out, actually is worked out by a play of forces,—spiritual, mental, vital and physical forces—and in that plane of forces there is no absolute rigidity discoverable. Personal will or endeavour is one of those forces. Napoleon when asked why he believed in Fate, yet was always planning and acting, answered, “Because it is fated that I should work and plan”; in other words, his planning and acting were part of Fate, contributed to the results Fate had in view. Even if I foresee an adverse result, I must work for the one that I consider should be; for it keeps alive the force, the principle of Truth which I serve and gives it a possibility to triumph hereafter so that it becomes part of the working of the future favourable Fate, even if the fate of the hour is adverse. Men do not abandon a cause because they have seen it fail or foresee its failure; and they are spiritually right in their stubborn perseverance. Moreover, we do not live for outward result alone; far more the object of life is the growth of the soul,—not outward success of the hour or even of the near future. The soul can grow against or even by a material destiny that is adverse.

Surrender to the Divine

Finally, even if all is determined, why say that life is, in Shakespeare’s phrase or rather Macbeth’s, “a tale told by an idiot full of sound and fury, signifying nothing”? Life would rather be that if it were all chance and random incertitude. But if it is something foreseen, planned in every detail, does it not rather mean that life does signify something, that there must be a secret Purpose that is being worked up to, powerfully, persistently, through the ages, and ourselves are a part of it and fellow-workers in the fulfilment of that invincible Purpose?
P.S. Well, one of the greatest ecstasies possible is to feel oneself carried by the Divine, not by the stars or Karma, for the latter is a bad business, dry and uncomfortable—like being turned on a machine, ‘yantrarudhani mayaya’.
[SABCL, 22:469-70]

Why blame the circumstances, one’s fate, one’s luck etc.?

In ordinary life this happens all the time. Only, you know, in ordinary life one says, “It is circumstances, it is fate, it’s my bad luck, it is their fault”, or else, “I have no luck.” That is very, very, very convenient. One veils everything and expects… yes, one has happy moments and then bad ones, and finally—ah, well, finally one falls into a hole, for everybody tumbles over, and expects to, sooner or later. So, one does not worry, or worries all the time—which comes to the same thing. That is, one is unconscious, one lives unconsciously and puts all the blame for what happens on others and on the circumstances but never tells oneself: “Why! It is my own fault.”… It needs a sufficiently vast consciousness to begin. Even among those who profess to be conscious, there are not many who see clearly enough to become aware that all that happens to them comes from what they are and from nothing else. They always say, “He is wrong; circumstances are unfavourable; oh! Why was that done?”—If you were not what you are, it would not happen in this way. It would happen differently.
[CWM2, 6:97]